CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation
Ingo Ziegler Abdullatif Köksal Desmond Elliott Hinrich Schütze

Estimating the coherence of noise in mid-scale quantum systems
    Inés de Vega
    Received ; accepted 
==============================================================

§ ABSTRACT 
Building high-quality datasets for specialized tasks is a time-consuming and resource-intensive process that often requires specialized domain knowledge.
We propose Corpus Retrieval and Augmentation for Fine-Tuning (CRAFT), a method for generating synthetic datasets, given a small number of user-written few-shots that demonstrate the task to be performed.
Given the few-shot examples, we use large-scale public web-crawled corpora and similarity-based document retrieval to find other relevant human-written documents.
Lastly, instruction-tuned large language models (LLMs) augment the retrieved documents into custom-formatted task samples, which then can be used for fine-tuning.
We demonstrate that CRAFT can efficiently generate
large-scale task-specific training datasets for four diverse
tasks: biology question-answering (QA), medicine QA and commonsense QA as well as summarization.
Our experiments show that CRAFT-based models outperform or achieve comparable performance to general LLMs for QA tasks, while CRAFT-based summarization models outperform models trained on human-curated data by 46 preference points.










§ INTRODUCTION


Large language models (LLMs) demonstrate strong
generalization capabilities across diverse tasks
<cit.>, but optimizing these models for specific tasks remains a
considerable challenge. Although zero-shot and few-shot
prompting methods provide some degree of adaptability
<cit.>, task-specific
fine-tuning generally delivers better performance,
particularly for specialized and out-of-domain tasks
<cit.>. A key challenge for effective fine-tuning
is obtaining high-quality task-specific examples at large scale.

Traditionally, creating high-quality datasets for specific tasks involves a time-consuming and resource-intensive process, often requiring extensive manual curation and annotation (e.g. <cit.>). This challenge is particularly acute for low-resource domains or novel tasks where existing datasets may be limited or non-existent.

On the other hand, “raw” (i.e., unannotated, free-text) web-crawled corpora are
known for their diversity and potential utility for various
tasks <cit.>. Prior work has used raw data by targeted crawling of recipe websites <cit.> or word-specific filtering of crawling metadata to gather examples from pre-training corpora for sentiment analysis and summarization tasks via ratings <cit.> and bullet point summaries found in news articles <cit.>.
These approaches either rely on a predefined task definition
based on keywords, or on the targeted crawling of websites
which are expected to contain the desired content.
This reliance hinders the generalization of these methods to tasks where such prior knowledge is unavailable, difficult to define, or highly context-dependent.

In this work, we propose Corpus Retrieval and
Augmentation for Fine-Tuning (CRAFT) to curate task-specific
samples from raw data for a wide variety of tasks. CRAFT
only requires a small set of few-shot examples from a
user to initiate the process of
crawling and structuring task examples. CRAFT
first detects relevant corpus examples from
large-scale unannotated corpora using similarity-based
retrieval. Then it uses LLMs to structure these examples
into a proper task format, effectively transforming
free-text documents into custom-formatted task samples
for fine-tuning.

We demonstrate the effectiveness of CRAFT on four diverse
tasks: three
QA tasks – in biology, medicine and
commonsense –  as well as a text
summarization generative task. Our results show that models
fine-tuned on CRAFT-generated datasets achieve performance
that is either better than or comparable to
instruction-tuned LLMs. This holds across diverse tasks,
LLMs, and dataset sizes, highlighting the effectiveness of
our approach. We publicly release the code to craft datasets
for other tasks as well as all datasets and checkpoints at
https://github.com/ziegler-ingo/CRAFTgithub.com/ziegler-ingo/CRAFT.





§ RELATED WORK



 §.§ Optimizing LLMs for Specific Tasks



  
Prompting:
Prompts are added to the input to provide additional context that guides the computation and output of a model <cit.>. A prompt usually takes the form of a natural language instruction <cit.>.
Prompting is commonly used with instruction-tuned models to define tasks and extract responses from language models, using natural language, without gradient updates.



  
Zero-Shot Inference:
Originally discovered in the vision domain, zero-shot inference <cit.> is a technique that allows models to generalize their learned knowledge from pre-training to previously unseen classes, tasks, or sample instance variants at inference time without gradient updates.
Pre-training LLMs on large corpora produces semantic representations that are generally applicable to multiple downstream tasks.
GPT-2 <cit.> demonstrated that the
acquired capabilities can then be activated by prompting a new task in natural language.
However, zero-shot inference often falls short of the performance achieved by few-shot learning <cit.>.



  
Few-Shot Learning:
In few-shot learning, the model is provided with a small
number of task-specific examples at inference time. 
The few-shot examples are given to the model in the prompt, in a technique known as in-context learning <cit.>.
While full fine-tuning generally requires a substantial amount of labeled data, few-shot learning offers an inexpensive alternative to adapt a model to a new task with a limited number of examples <cit.>.
Nonetheless, few-shot learning faces several challenges,
including inaccurate assessment of the underlying data
distribution <cit.>, biases related
to small sample sizes <cit.>, and
sensitivity to shot length <cit.>, shot quality and noise <cit.>.



  
Full Fine-Tuning:
During full fine-tuning, all model parameters are updated on
a large dataset with the goal of
adapting the model to a domain, task or dataset <cit.>.
This approach usually provides the best performance by learning task-specific patterns and relationships that may not be captured by pre-training and zero- or few-shot learning alone.
However, it requires a dataset of appropriate size.



  
Instruction Tuning:
Instruction tuning <cit.> is a type of full fine-tuning that optimizes a model to produce more relevant answers to questions or instructions <cit.>.
This approach enables language models to better understand and follow user intents rather than simply continuing the input text.
Instruction-tuned models regularly produce answers that are preferred by humans for tasks ranging from question-answering to summarization <cit.>.
The main challenge is to obtain a large high-quality dataset that is both task-specific and in the desired instruction-output format.




  
Low-Rank Adaptation:
Full fine-tuning may be too expensive for LLMs but the
difference between pre-trained weights and their fine-tuned
counterparts often has low rank <cit.>.
Low-Rank Adaptation <cit.> approximates these low-rank matrices during fine-tuning, and is efficient because it freezes the full model and only learns the low-rank matrices, which typically results in learning the equivalent of 2% of the model's parameters.



 §.§ Synthetic Data Generation

Synthetic data refers to artificially generated data that mimics the characteristics of real-world data <cit.>.
It can be generated using statistical <cit.> or deep neural approaches <cit.> with the aim of replicating the patterns, distributions, and structures found in real-world datasets.



  
Fully Synthetic Data Generation:
A dataset is fully synthetic if the question or instruction, the possible context, as well as the answers are generated synthetically.
For instance, Self-Instruct <cit.>, Unnatural
Instructions <cit.>,
Alpaca <cit.>, and
Evol-Instruct <cit.> are examples of fully synthetic general-purpose data generated by LLMs.
More focused approaches for task-specific fine-tuning data generation have also been proposed, especially based around the rephrasing of already existing task-specific datasets <cit.>.
Methods that use general-purpose corpora have recently been proposed for generating pre-training data <cit.>.
When used for fine-tuning data generation, these methods are either based around complex and resource-intensive multi-agent workflows <cit.> or are restricted to a small set of tasks, as the generation process relies on a model that has been fine-tuned for those tasks <cit.>.

The two greatest drawbacks of current approaches to fully
synthetic data generation are repetition and low quality.
Unnatural Instructions reported that a majority of
their samples have a BERTScore <cit.> of
above 45% when compared to other samples in the generated
dataset. Self-Instruct faces similar issues, with generated
instructions often having ROUGE-L
scores <cit.> greater than 0.4 compared to
the provided seed instructions. Both approaches also only
contain about 54%-56.5% correct samples, while the
correctness rate in Alpaca is as low as
17% <cit.>. This suggests that a large
portion of the samples in these datasets may not be useful
for fine-tuning models.






  
Partially Synthetic Data Generation:
In partially synthetic data generation, a portion of the input, context, or output is generated synthetically, while the remaining portion is human-curated.
It is distinct from approaches that combine fully synthetic and purely human-curated samples at the dataset level, such as Phi <cit.>.

One recent approach is reverse instruction
generation <cit.>, where a language
model,
provided with a human-curated output in context,
generates
the instruction that would have prompted this output.
This produces more coherent and
correct input-output pairs because the LLM does not need to
generate the longer and more complex component of the data
sample. There are also approaches where, conversely, the
output is synthetically generated from human-curated input
samples. Such methods employ distillation techniques to
extract patterns from larger, more capable models to teach
those patterns and skills to smaller
models <cit.>.

Partially synthetic data generation alleviates some of the
quality and diversity concerns of fully synthetic data generation.
However, taking a raw corpus document as the output can result in generating noisy or unnecessary information <cit.>, but data augmentation can mitigate these problems when generating pre-training data <cit.>.
However, when data augmentation was used to generate fine-tuning data, it required GPT-4 <cit.> to build an intermediate synthetic dataset to fine-tune a sample creator model <cit.>.
This approach can result in a sample creator model that is a distilled version of the larger model's knowledge and data, while also limiting the model's task flexibility, depending on the synthesized training data.

In contrast, CRAFT produces fully synthetic data but leverages the quality and diversity advantages of human-written documents from partially synthetic data generation approaches while removing noise through augmentation.
Our approach does not require intermediate datasets, nor a
separately fine-tuned model, nor knowledge distillation from
a larger model; instead, it relies only on a small number of human-curated examples, retrieval, and in-context learning.




§ THE CRAFT APPROACH



 §.§ Architecture Overview

CRAFT is used to fine-tune language models by generating task-specific synthetic datasets, given a few human-curated examples of the task.
During CRAFT (see Figure <ref>), we retrieve
human-written, free-text documents from a large collection
of corpora by calculating their similarity to the provided
few-shots and transforming them into
the task-specific format
through augmentation.
The only human effort required is in writing a small number of high-quality examples of the target task.
CRAFT has two phases: In the initial phase, an embedding database is created from large corpora. 
While this phase can be resource-intensive, its cost is incurred only once for all subsequent tasks, and it can be easily expanded with new corpora.
In the second phase, the user-generated, task-specific few-shot examples are embedded,
enabling the retrieval of relevant documents by calculating similarity measures between few-shots and corpus documents.
Once relevant documents are retrieved,
an instruction-tuned LLM is used to
augment the retrieved free-text documents into a
task-specific design, generating synthetic task samples in
the layout that is needed for instruction-tuning (illustrated in Figure <ref>).
Finally, the synthetic dataset is used to fine-tune a task-specific language model.
We report implementation details for the whole CRAFT framework in Appendix <ref>.








 §.§ Few-Shot Examples

A small number of human-curated few-shots serve as the
“definition” of the task, i.e., they indicate how the task
is to be performed.
The few-shot samples consist of three elements:
(i) a long text that mirrors in language, content, and accuracy what a high-quality corpus sample from the web should look like,
(ii) a natural language instruction for the task to be performed, which can take the form of a direct instruction or a question about the text, and 
(iii) an output that satisfies the instruction or answers the question the way that the final model should later respond.
Length statistics for texts, instructions, and outputs of our few-shots can be found in the XS row of Appendix <ref>.

We note that the task does not need to be explicitly
specified. For example, there is no need to state the task
as “biology question-answering”; it is sufficient for the
human-curated few shots to focus on QA in the domain of
biology.
If multiple-choice questions or single-letter outputs are in the few-shots, this will result in a corresponding dataset and fine-tuned model behavior.
These examples show that CRAFT is highly customizable: Few-shot
examples enable users to tailor the model’s behavior to
specific formats, use cases, or domains. Users can create
few-shots with unique terminology, style preferences, or
domain-specific constraints, optimizing the retrieval and
the final model’s performance for particular tasks.



 §.§ Corpora and Embedding Database

The embedding database is a key element of CRAFT as it
provides, for all corpora, embeddings of human-written documents that should be retrievable for task-specific augmentation.
It is, therefore, important that the embedding database encompasses a wide variety of linguistically and semantically diverse documents.
This diversity can be achieved by including corpora that exhibit different writing styles, tones, and vocabularies.
Task-specific, task-agnostic, public, and also private documents can provide a comprehensive coverage of relevant information.
The more varied the documents in the embedding database, the
better the coverage
will be for diverse or rare tasks.
Notably, CRAFT can also handle sensitive company data, as the encoding, storage, and retrieval can be performed on-site.



 §.§ Document Retrieval


Our retrieval system is task agnostic, both in terms of domain and complexity, in contrast to previous approaches <cit.>.
The CRAFT approach relies on human-curated few-shot examples as query documents and can dynamically retrieve any document of the base corpora.
As the few-shot samples include a text containing the domain, the instruction or question, as well as the output, the resulting embedding representation of the sample contains contextualized <cit.> semantic information about both the domain and the nature of task to be performed.
Relevant text documents that contain similar latent features as the few-shots are retrieved from the corpora by calculating similarity scores based on the embedded few-shots and corpus samples.

As corpus size increases, the risk of retrieving redundant or similar corpus samples also increases.
This is partly due to the growing volume of documents, but also because the diversity of documents within the corpora may plateau, resulting in a higher proportion of similar documents. Designing few-shots that are sufficiently diverse in topic may alleviate this issue.
For example, when creating few-shots for biology question-answering, various subtopics of biology, such as genetics, anatomy, or physiology, should be covered to broaden the range of retrieved documents.



 §.§ Task Sample Synthesis

The retrieved documents naturally contain noise <cit.> and lack the formatting required for fine-tuning.
Therefore, it is necessary to convert these free-text
documents into appropriate task samples
by removing noise and undesired sections.

To address this, we utilize instruction-tuning prompt templates <cit.> to augment the free-text documents into task-specific training data while simultaneously eliminating noise.
A few-shot task template consists of three elements:
(i) one or more few-shots,
(ii) a corpus sample,
(iii) and a brief instruction for the model to generate instruction-output pairs from the content of the corpus sample.
Aside from the brief instruction,
it is easy to assemble 
these templates
from material we already have.
The template only structures all information from the instruction, the few-shots, and the retrieved corpus samples to generate one continuous string that serves as input for the model generating the synthetic task samples. 
In this setup, the contents of the few-shots serve as in-context examples for the completion of the instruction.
Figure <ref>, step 3, shows an example of how these templates guide the model in augmenting the corpus samples into synthetic task samples.
Any instruction-tuned language model can be used for this purpose.
This augmentation step not only rephrases the text but also condenses the retrieved document down to the essential information required for the task.
The result of this step produces the final synthetic instruction-output pairs that can be used to fine-tune a language model.
Figure <ref>, step 4, shows an actual example output from the generated pool of synthetic training samples, and Appendix <ref> provides an overview of length statistics from the stages of corpus retrieval up to the synthesized input-output pairs.



§ EXPERIMENTAL SETUP



 §.§ Tasks

We generate datasets for all tasks in sizes 100, 500, 5000, and 25,000.
We refer to the few-shots as a dataset of size XS, and to the sizes ranging from 100 to 25,000 as S, M, L, and XL, respectively.
Implementation details related to fine-tuning can be found in Appendix <ref>.



  
Multiple-Choice QA:
We generate three synthetic QA datasets for biology (BioQA), medicine (MedQA), and commonsense (CSQA).
The datasets all follow the MMLU multiple-choice format <cit.>, where each question is accompanied by a number of answer options.
Exactly one of these options is correct, and the task is to identify the correct answer.
The expected output is the corresponding letter label of the correct answer.



  
Generative:
In addition, we develop two synthetic datasets for the generative tasks of summarization and recipe generation (RecipeGen).
The goal of summarization is to convey accurate and concise information; recipe generation is additionally focused on creating coherent and structured text that adheres to specific formatting and stylistic conventions <cit.>.
To build a synthetic summarization dataset, we first select a corpus sample and instruct the model to extract an extensive but unsummarized section of text.
In the second step, the extracted section is transformed into a summary format, optionally incorporating elements from the raw text, such as abstracts, conclusions, or TLDRs.
This approach avoids using the original corpus samples as to-be-summarized text documents, which can be lengthy and overly broad and may result in uninformative summaries. 



 §.§ Evaluation



  §.§.§ Metrics




  
QA Tasks: 
All multiple-choice QA tasks are evaluated using accuracy.
We follow the evaluation approach of MMLU and assess the logarithmic probabilities for the vocabulary options corresponding to the letter labels of the answer choices <cit.>.
Accordingly, greedy decoding without temperature scaling is performed.
Depending on the number of answer choices, this evaluation can range from options A and B to options A through E.



  
Generative Tasks:
Automated metrics like ROUGE <cit.> and METEOR <cit.> are resource-efficient as evaluation metrics but face limitations in generative tasks.
They rely heavily on n-gram overlap, which may not accurately reflect the true quality of the generated text <cit.>.
They also assume that the reference text provides a complete and accurate representation of the desired output, which is not always guaranteed <cit.>.
Reference texts can be low-quality, contain errors and ambiguities, which leads to unreliable evaluation results. This issue is illustrated by the findings of <cit.>, who demonstrated that human reviewers frequently rate gold-standard benchmark answers among the worst answer options.
This issue worsens when generated texts differ significantly in length from references, as subsequence-based metrics struggle to capture such variations <cit.>.

As an alternative, we opt to evaluate generations using LLMs as a judge <cit.>.
In this setup, the LLM effectively acts as a human annotator, providing a binary preference score for each pair of outputs, resulting in a win rate as the final metric <cit.>.
This approach has become increasingly common: LLMs have been successfully employed as annotators while demonstrating high inter-rater reliability <cit.>. 

For general-purpose outputs, we use the popular Alpaca-Eval
benchmark <cit.>
that evaluates
multiple LLMs on about 650 human-curated questions <cit.>. We select Llama 3 70B <cit.> as our annotator model due to its open nature and cost-efficiency for high-volume experiments. As of July 2024, Llama 3 70B ranks 4th in human agreement with a score of 67.5, close to customized GPT-4 versions at 69.2.





  §.§.§ Datasets

To assess the quality of our synthetically generated datasets, we compare the performance of models trained on them to those trained on human-curated datasets.



  
BioQA:
We use the 800 sample test split from the biology subsection of ScienceQA <cit.>.
ScienceQA sourced expert-curated question-answer pairs from online learning platforms, ensuring a high level of quality and accuracy.
The dataset's answer options range from two to five, have a single correct answer per question, and are randomized to prevent pattern recognition.



  
MedQA:
We use the 4183 samples from the MedMCQA validation split <cit.>.
The dataset is comprised of entrance exam questions to two of India's postgraduate institutions.
All dataset samples are sourced from preparation or real exams created by medical professionals.
All questions have 4 answer options with one correct answer option per question.




  
CSQA:
We benchmark on the 2541 validation set samples in CommonsenseQA 2.0 <cit.>.
The dataset was generated through a gamified but controlled question generation process, where players were incentivized to design challenging yes/no questions by earning points when beating an AI model.
The generated questions were then cross-validated by other players, independent validators, and another model, to ensure that the questions were well-formed, answerable, and representative of common sense.



  
RecipeGen:
We randomly sample 1000 samples from the higher-quality subsection of the RecipeNLG dataset <cit.>.
The recipes were scraped from cooking websites and post-processed using a fine-grained and standardized cleaning and formatting process to ensure correctness.
Each recipe features a title, the list of required ingredients, as well as the steps to produce the meal.
We only include samples not present in C4, based on the URL in each dataset.







  
Summarization:
We use 1000 samples from the test split of the CNN-DailyMail dataset <cit.>.
The dataset is commonly used for summarization because it consists of articles and stories from CNN and DailyMail alongside their highlights in abstract or bullet point format presented at the top of newspaper pages or websites.



§ RESULTS



 §.§ Baselines

We compare the performance of CRAFT, trained on our synthetic datasets, against three baselines.
The few-shot baseline is a model fine-tuned only on the XS size CRAFT dataset, with human-curated few-shots.
It serves as the primary baseline since this model uses all
human-curated
data available in our pipeline.

The second baseline is the instruction-tuned model, Mistral 7B Instruct v0.2 <cit.>,
which has been fine-tuned on proprietary instruction-following datasets that mix various tasks and sources.
This baseline provides a meaningful comparison, as it is similar in size and instruction-tuned like CRAFT models, though it is trained on undisclosed datasets of unknown quality and quantity.
Thus, matching or exceeding the performance of instruction-tuned models with our synthetic data would indicate that CRAFT can produce high-quality datasets.

The upper bound of expected performance is fine-tuning the models on the in-domain human-curated training splits from the chosen evaluation datasets.
This baseline represents the optimal performance achievable with human-quality datasets.



 §.§ Scaling the Data


We report the performance gain when scaling up our training data in Figure <ref>. 
We report the mean
and standard deviation
across three seeds.
We observe consistent improvements across four tasks as we
increase the data size. Relative to the baseline models
trained with only few-shot examples, we see improvements of
17% (from 66.9 to 78.1), 12% (from 55.3 to 62.1), 23%
(from 39.1 to 48.0), and 124% (from 43.7 to 97.9) for
BioQA, CSQA, MedQA, and Summarization, respectively. This
shows that CRAFT can be used for diverse tasks, starting
with just a few curated examples. We also find
appropriate scaling for each set of examples, ranging from
100 to 25,000 across all tasks. Additionally, 
that models trained with fewer examples (32, 100) exhibit
much more variance than those trained with 5,000 and 25,000
examples, as indicated by the gray regions in the plots that visualize the standard deviation.






For all tasks, we achieve 
results
that are are clearly better of comparable to
Mistral Instruct. It is worth
noting that CRAFT uses an LLM in a limited way (to
restructure and rewrite existing corpora) that seem to
exclude the possibility that distillation may have played a
role here. However, even if distillation were to be considered the
reason for good CRAFT performance, the results indicate
otherwise:
we use the same model (Mistral
Instruct) to paraphrase existing corpora examples but
achieve even stronger results. Finally, we observe that
CRAFT models outperform those trained with official
human-curated large-scale data in Summarization. For other
tasks, while we observe lower performance than with official
data, we speculate that this could be due to in-domain
evaluation for official human-curated data. We use their
test split to evaluate our models, which may give these
models an unfair advantage. We investigate this further in
the next section.






 §.§ OOD Generalization and Data Contamination

We now report experiments to understand the level of data contamination or similarity between test and training examples in the experiments introduced in Figure <ref>. We conduct 5-gram weighted Jaccard similarity analyses between CRAFT datasets and the test data.
For each sample, we combine the instruction and output and gather 5-gram frequencies for the whole dataset.
We then calculate the Jaccard similarity between the 5-gram
frequency distributions of the respective CRAFT and test
dataset, where 5-grams receive weight proportional to their frequency.

This analysis shows that all CRAFT datasets have less than
0.4% similarity with the task test set, whereas the
original human-authored datasets show much higher
similarities: BioQA (17.9%); CSQA (4.4%), MedQA (1.1%),
and Summarization (0.3%); this may indicate some overlap
between train and test splits. Since we have curated the
few-shots manually, rather than copying from
existing datasets, our low overlap (0.4%) may be expected. To
further investigate CRAFT's performance improvement, we
select four out-of-domain datasets for the biology question
answering task and compare CRAFT with the human-curated
baseline.






In Table <ref>, we compare the human-curated baseline with CRAFT on the in-domain dataset for the baseline and four out-of-domain (OOD) datasets selected from MMLU <cit.>. Although the baseline outperforms CRAFT by more than 11% in the biology subset of the ScienceQA test set (in-domain for the baseline), CRAFT outperforms the baseline on 3 out of 4 OOD datasets. On average, CRAFT outperforms the baseline by 4%.











 §.§ Negative Results: Recipe Generation

Out of the five tasks we selected, we observe non-scaling behavior in one task: Recipe Generation. While our manually curated few-shots are of high quality, we see a drop when scaling from 32 to 25,000 examples, as illustrated in Figure <ref>. CRAFT's performance is still better than the baseline with official human data, which means that the final dataset is usable. However, we explore why this reverse scaling occurs and examine the drop in performance.




An initial analysis suggested that the CRAFT
pipeline tends to find less relevant examples over time. We conducted automated data
quality analysis to
analyze this on a larger scale. For 500 randomly sampled instructions from
different sizes of CRAFT datasets (i.e., the training sets), we used the
Llama 3 8B Instruct model to answer the instructions. Then,
using Llama 3 70B Instruct as a judge, we compared win
rates, i.e., which output the model preferred: the gold
output in the CRAFT dataset or the output generated by
Llama 3 8B Instruct. We report the average win rate against the Llama outputs as the data quality metric. Higher scores
indicate that the pipeline created higher quality output
than Llama 3 8B Instruct's answers.

We observe that data quality drops when scaling up to
25,000 examples. The 100 and 500 example sets have win rates
around 0.4, while the win rate for 25K drops to
0.3. We believe this is the cause of the performance drop
with scaling. While the final dataset
is still useful (it outperforms the baseline with
official human data), the next version of CRAFT should
include either effective stopping criteria or additional
validators for quality.




 §.§ Base Model Comparison

In previous sections, we fine-tuned CRAFT models using the pretrained Mistral 7B model. Now, we repeat the experiments using the pretrained Llama 3 8B model. We observe similar trends across all tasks, and the relative improvement is comparable when scaling up from few-shots to 25,000 examples, as illustrated in Table <ref>.


In all experiments, we manually curated 32-shot examples and expanded our synthetic data examples from that point. However, even curating 32 examples can be time-consuming. We can limit the number of few-shots to just eight examples to bootstrap the CRAFT process. Figure <ref> shows the results compared to the CRAFT pipeline with 32-shots. While the final results with 25,000 examples are slightly lower for 8-shots, the trend is similar for both 8- and 32-shot examples. We observe that the model trained with 25,000 samples, based on running CRAFT with 8 few-shots, significantly outperforms the model trained with only 32 few-shots (i.e., no extra synthetic data). This suggests that if there are time and resource limitations, using CRAFT with fewer initial examples leads to better models than trying to curate and train only on more human-curated few-shot examples.



§ CONCLUSION

In this work, we introduced CRAFT (Corpus Retrieval and
Augmentation for Fine-Tuning), a framework for generating
task-specific synthetic datasets grounded in
text
corpora. CRAFT requires only a small set of human-curated few-shot examples to bootstrap the creation of large-scale training data by leveraging existing corpora and instruction-tuned language models. Our experiments across multiple tasks, including biology, medicine, and commonsense question-answering, as well as summarization, demonstrate that models fine-tuned on CRAFT-generated datasets can match or outperform strong baselines, including instruction-tuned models and those trained on human-curated datasets. Notably, CRAFT-based models showed better generalization capabilities on out-of-domain datasets compared to models trained on human-curated data, highlighting the robustness of our approach.

While CRAFT shows promising results for most tasks, we also identified limitations in scaling performance for recipe generation, emphasizing the need for careful quality control and potential stopping criteria in future iterations. Nevertheless, the overall success of CRAFT in producing high-quality synthetic datasets with minimal human effort opens up new possibilities for efficient and adaptable model fine-tuning across a wide range of domains and tasks.



§ ACKNOWLEDGEMENTS

IZ and DE have been supported by the European Union’s Horizon 2020 research and innovation program under grant agreement No. 101135671 (TrustLLM).
AK and HS have been funded by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) project SCHU 2246/14-1.
IZ was also supported by a G-Research Travel Grant to present this work at the ELLIS Doctoral Symposium 2024 in Paris.


acl_natbib








§ IMPLEMENTATION DETAILS




 §.§ Few-Shot Design

The BioQA few-shot text samples were compiled from a diverse range of sources, including textbooks <cit.>, openly accessible materials, and Encyclopedia Britannica.
For MedQA, we primarily drew upon openly accessible websites, such as the National Institutes of Health, National Health Service, Food and Drug Administration, Mayo and Cleveland Clinics, and other medicine-related websites, to generate our few-shot text samples.
The CSQA few-shot text samples were sourced from a variety of online resources, including blogs, articles, and other websites tailored to the specific topic at hand.
From sources that are not openly accessible through websites, continuous text snippets were directly extracted and used as texts, while all other material was shortened, rephrased, or restructured by the authors.
This process ensures that articles which may have been crawled through C4 <cit.> do not produce exact matches at retrieval time.
Since none of these texts have direct question-answer pairs available, the question, answer options, and the answer were generated by the authors for each sample.
Table <ref>, step 1, shows an example.

The recipe generation few-shots were taken from blogs and other openly accessible websites.
Their design always includes a simple instruction or question to cook a meal, as well as bulleted or numbered lists of ingredients and steps to produce the meal.
Sometimes, the recipes on websites were described only as continuous text.
In such cases, the authors added the list of ingredients and steps to create the few-shots.
Similar to the QA few-shots, it is desirable to increase the vocabulary diversity for retrieval.
Therefore, we made sure to cover a wide range of recipes in the few-shots.
To design summarization few-shots, we collected a wide variety of texts from websites, blogs, and magazines, as well as specialized sources such as GitHub issues.
Here, the few-shots feature a text, an instruction to create a summary, as well as the summary output.
On some websites, summarized versions of the main text are given, while in other cases continuous or bulleted summaries were created by the authors.



 §.§ Embedding Database

To create the embeddings, SentenceTransformers <cit.> is used, specifically a MiniLM <cit.> version () optimized for cosine-similarity search between two document pairs.
This model creates 384D embeddings,
which are stored in an HDF5 database <cit.>, that allows for native storage and retrieval of array-like data.



 §.§ Corpora

To enable retrieving human-written documents for general-purpose as well as specialized domains, we include four large corpora.

The Colossal Clean Crawled Corpus <cit.> dataset consists of approximately 750GB of English data, pre-cleaned by the creators to exclude non-informative texts.
We use a 305GB subset of the dataset, which excludes not-safe-for-work or offensive content.

The English Wikipedia corpus comprises a diverse and high-quality collection of textual information on a wide range of topics.
We use the January 2nd, 2024 dump of English Wikipedia, which we pre-process using WikiExtractor <cit.> to extract clean text documents.

The Stack Exchange corpus <cit.> features a structured format with title, body, and best-voted answer collected from the Stack Exchange network.
The dataset encompasses 173 distinct sub-communities, covering both technical and non-technical topics.

The WikiHow corpus <cit.> presents information in a step-by-step instructional layout, making it a valuable resource for tasks such as summarization or recipe generation.
Each document consists of a title, clear instructions, and accompanying text.

After filtering out documents with fewer than 200 characters or more than 25,000 characters, the resulting datasets contain approximately 362 million documents from C4, 10.5 million documents from English Wikipedia, 9.5 million documents from Stack Exchange, and 190,000 documents from WikiHow.
The resulting 383 million documents take up approximately 247GB of storage when GZIP compressed <cit.> and represented as 16-bit NumPy arrays <cit.>.




 §.§ Document Retrieval

To retrieve relevant human-written samples, we employ a two-step process that approximates a global similarity search.
Due to the large size of the embedding database (383 million documents), a direct global pair-wise comparison is impractical.

In the first step, we divide the embedding database sequentially into subsections of approximately 350,000 documents each.
We calculate cosine similarity between each subsection and the few-shot samples, and store the top 5% most similar documents in memory.
This reduces the number of documents considered from 383 million to approximately 19 million.
In the second step, we recalculate cosine similarity and perform traditional top-k similarity retrieval on the filtered documents. 
This yields the indices of the top-k most similar documents to the few-shot examples.

Optimizing solely for top-k similarity between individual few-shot samples and the embedding database may lead to limited variation and high similarity among the retrieved documents.
Conversely, optimizing for similarity by averaging the embedding representations of all few-shot samples may prioritize a single dominant topic, potentially biasing the selection.
To balance these approaches, we adopt a mixed similarity retrieval strategy, combining two methods:
(i) 50% of samples are retrieved based on individual top-k similarity to each few-shot sample, ensuring a minimum number of similar samples for each few-shot and mitigating topic dominance, and
(ii) the remaining 50% of samples are retrieved based on fully averaged similarity, which aims to help uncover more latent topics.




 §.§ Task Sample Synthesis

To facilitate the creation of high-quality synthetic task samples, we align the text generation process to the few-shot design through in-context learning <cit.>.
This approach helps reduce issues like hallucinations <cit.> and formatting errors, although it does not entirely prevent them.
Specifically, three few-shot examples from the human-curated few-shots are dynamically and randomly sampled for each input prompt and corpus sample, interleaving them with the instruction to augment the text.
Since each few-shot example includes a text similar in length to corpus samples, the three-shot prompting technique can result in input prompts that often exceed 10,000 tokens, sometimes reaching 20,000 tokens.
For optimal performance, it is recommended to use models offering long context lengths.

In our experiments, all task samples are created using Mistral 7B Instruct v0.2 <cit.> along with the few-shot task template shown in Table <ref>, step 3.
During the generation process, vLLM <cit.> is used with temperature 0.7, top-k <cit.> and top-p <cit.> sampling at 40 and 0.9, respectively. 
The maximum generation length was determined based on preliminary experiments with the goal of accommodating full-length sample generation.
For all QA datasets we limited samples to 256 tokens, whereas recipe generation and summarization were limited at 1280 and 1536 tokens, respectively.

To enable quality control checks such as checking whether the QA datasets have the desired number of answer options, or whether the samples have been fully generated, we instruct the model to produce all task samples as valid JSON objects with a set number of keys and proper naming.
We perform checks on each JSON object to ensure that the questions, instructions, and generative task answers exceed a reasonable minimum length, as well as that the answer option generated for the QA tasks contain valid letters.
If no valid JSON is found, or keys are missing, or any formatting check fails, we discard the sample.
Furthermore, we filter task samples that are either too similar to the few-shots or to other generated task samples using fuzzy string matching <cit.> at a token set similarity ratio of 0.85.
To account for the removal of samples during deduplication, we recommend retrieving approximately twice as many corpus samples as the number of desired final synthetic task samples.
A detailed breakdown of the retrieval and deduplication quantities for each task is provided in Appendix <ref>.




 §.§ Training and Optimization


For all experiments, low-rank adaptation <cit.> fine-tuning is performed with 16-bit BrainFloat <cit.> as the computation data type.
All experiments are implemented using PyTorch <cit.> and the Hugging Face packages <cit.>.
For optimization, the adaptive momentum optimizer with decoupled weight decay <cit.> of 5% and a learning rate of 1 × 10^-4 is employed.
A linear learning rate schedule is applied with a warm-up ratio of 10%.
Fine-tuning is performed for 3 epochs across all tasks and dataset size variations.
When fine-tuning on only the human-curated few-shots, a batch size of 2 is adopted.
In all other scenarios, fine-tuning is performed with a batch size of 16 or the equivalent with gradient accumulation steps.
Following <cit.>, LoRA adapters are used on every linear layer, specifically on the query-, key-, value-, output-, gate-, up-, and down-projection matrices.
We use a rank and α parameter of 64, the adapter's dropout rate is set to 0.1, and the bias terms of the update matrices are deactivated.
In total, this adds 2.3%, or about 160 million parameters, of the model's 7 billion base parameters as LoRA adapters.
The rest of the model remains frozen during fine-tuning, and the updated parameters will be merged with the base model after fine-tuning.





§ FILTERING STATISTICS















§ DATASET STATISTICS








