Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
Under review as a conference paper at ICLR 2025

TOWARDS A THEORETICAL UNDERSTANDING OF SYN-
THETIC DATA IN LLM POST-TRAINING:
A REVERSE-BOTTLENECK PERSPECTIVE

Anonymous authors
Paper under double-blind review

ABSTRACT

Synthetic data has become a pivotal resource in post-training tasks for large lan-
guage models (LLMs) due to the scarcity of high-quality, specific data. While
various methods have been developed to generate synthetic data, there remains
a discernible gap between the practical effects of synthetic data and our theo-
retical comprehension. To address this challenge, we commence by presenting
a detailed modeling of the prevalent synthetic data generation process. Build-
ing upon this modeling, we demonstrate that the generalization capability of the
post-trained model is critically determined by the information gain derived from
the generative model, as analyzed from a novel reverse-bottleneck perspective.
Moreover, we introduce the concept of Generalization Gain via Mutual Infor-
mation (GGMI) and elucidate the relationship between generalization gain and
information gain. This analysis serves as a theoretical foundation for synthetic
data generation and further highlights its connection with the generalization capa-
bility of post-trained models, offering an understanding about the design of syn-
thetic data generation techniques and the optimization of the post-training process.
We open source our code through an anonymous GitHub repository at https:
//anonymous.4open.science/r/Understanding-Synthetic.

1

INTRODUCTION

The efficacy of large language models (LLMs) is extensively influenced by both the volume and
quality of the training data, as established by the widely acknowledged scaling laws (Kaplan et al.,
2020). Given the inherent sparsity of data available during the post-training phases of LLMs, syn-
thetic data plays a critical role, particularly during fine-tuning and alignment processes. Over the
past decades, the LLM community has increasingly employed synthetic data to augment training in
scenarios where real data is scarce. As of September 2024, there are over 1,000 datasets labeled as
“synthetic” on the Hugging Face platform1. Several leading-edge large language models, includ-
ing LLaMA (Dubey et al., 2024), Falcon (Almazrouei et al., 2023), Qwen (Bai et al., 2023), and
GPT-4 (OpenAI et al., 2024), have also reported utilizing synthetic data during their post-training
stages. These instances underscore the pivotal role of synthetic data in enhancing the post-training
of LLMs.

Numerous methodologies for synthetic data generation have been advanced (Patel et al., 2024;
Møller et al., 2023; Park et al., 2024), yet the most prevalent and efficacious approach within the
community involves generating synthetic data through sampling from a proficiently trained gener-
ative model, often another LLM tailored for specific domain tasks. To delineate this process more
precisely, Long et al. (2024) describe the generation of synthetic data as follows: a well-trained gen-
erative model M is utilized, and synthetic data Sgen is produced by sampling from M , conditioned
on a set of prompts p, just as illustrated in the lower part of Figure 1 (a).

Synthetic data with such a generation manner is widely recognized and has been verified to be
effective in LLM post-training practice. However, several challenges persist that compromise its
potential benefits. First, the quality and diversity of synthetic data can vary significantly depending

1https://huggingface.co/

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: An overview of the synthetic data generation modeling and the relationships between the
distributions. (a) The synthetic data generation process and the corresponding distribution compres-
sion process. (b) The relationships between the distributions in the generation process.

on the generation method and the underlying model parameters (Koo et al., 2023). This variability
can lead to inconsistent training outcomes and may not fully address the sparsity in real data. Ad-
ditionally, while synthetic data offers a promising solution to enrich the limited real data, ensuring
that it sufficiently mimics real-world complexities without carrying over biases or errors from the
original data is still a daunting task (Villalobos et al., 2022). Addressing these challenges requires
a nuanced understanding of both the generation processes and their interaction with model training
dynamics.

Unfortunately, there remains a significant gap in the rigorous modeling of synthetic data, which
in turn limits a deeper understanding of its inherent mechanisms (Liang et al., 2024). This lack
of a comprehensive theoretical framework hinders our ability to predict the effectiveness of syn-
thetic data across different LLM applications and constrains the optimization of generative models
for more targeted data synthesis (Giles et al., 2022). Consequently, advancing our knowledge on
how synthetic data interacts with LLMs during training phases is crucial for enhancing model per-
formance and reliability, and can enable the development of tailored synthetic datasets that more
effectively address specific gaps in training data, thereby enhancing the overall performance and
generalization capabilities of large language models.

In this paper, we endeavor to examine the influence of synthetic data on the post-training phases of
large language models (LLMs) through an analytical lens focused on data distribution and informa-
tion content. Our investigation seeks to address the following theoretical questions:

• What underlies the effectiveness of synthetic data? How can we model the data generation

process and connect it with the generalization capabilities of post-trained models?

• What is the reason for the effectiveness of synthetic data in LLM post-training?

In response to these inquiries, we introduce a theoretical framework designed to dissect the impacts
of synthetic data on LLM post-training. The principal contributions of our study are outlined as
follows:

1. We develop a modeling of synthetic data generation from a distributional perspective, pro-
viding a theoretical foundation for understanding the generation process and its implica-
tions on LLM post-training.

2. Drawing on this modeling, we propose a reverse-bottleneck framework that elucidates

the mechanisms through which synthetic data influences LLM post-training.

3. We perform a theoretical analysis from an information-theoretic standpoint, delivering sev-
eral upper bounds that quantifies the expected generalization capabilities of LLMs when
trained with synthetic data.

2

𝑺𝒂𝒏𝒄𝒉𝒐𝒓Prompt𝑝Output𝑀𝜙𝒯𝒟𝒟"𝒟!(⋅|𝑝)𝑺𝒈𝒆𝒏+𝜀𝒟&’(Distribution CompressionSynthetic Data Generation(a) Overview of synthetic data generation(b) The relationships between distributions𝒟)𝒟𝐷"#(𝒟!,𝒟$%&)𝒟’()Data Curation+𝜀Generation DivergencePrompting𝑀(𝑝)𝒟)(⋅|𝑝)Task Divergence𝐷"#(𝒟!,𝒟)Under review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

The remainder of this paper is structured as follows. In Section 2, we provide a comprehensive
review of literature pertinent to our research. In Section 3, we first delineate the symbols and foun-
dational concepts critical to our analysis, then introduce the modeling for synthetic data generation
and bridge its connection with generalization capability of post-trained models. Section 4 introduces
our novel reverse-bottleneck framework, designed to assess the effects of synthetic data on post-
training stages of LLMs, and to establish generalization error upper bounds. The paper concludes
with Section 5, summarizing our findings and discussing potential avenues for future research.

2 RELATED WORK

2.1 GENERATIVE DATA AUGMENTATION

Generative models constitute a category of machine learning models that are specifically trained to
create new data points mimicking the original data distribution. Various types of generative models
have been developed, each suited to particular data types and model architectures. Notable among
these are Variational Autoencoders (Kingma, 2013), Generative Adversarial Networks (Goodfellow
et al., 2014), Normalizing Flows (Rezende & Mohamed, 2015), and, more recently, diffusion mod-
els (Rombach et al., 2022). Building on this premise, generative data augmentation has emerged
as a promising approach to bolster machine learning model performance (Yamaguchi et al., 2020).
This technique involves scaling up the available training dataset by generating new data points from
a limited pool of labeled data using generative models. Empirical evidence suggests that generative
data augmentation is particularly effective across various tasks, including knowledge graph reason-
ing (Maharana & Bansal, 2022), text-to-image generation (Yin et al., 2023), and relation extraction
from natural language texts (Hu et al., 2023). Theoretical investigations have also been conducted
to elucidate the underlying mechanisms through which generative data augmentation delivers these
benefits (Zheng et al., 2023a). Collectively, these advancements highlight generative data augmen-
tation as a highly promising avenue for improving machine learning model performance, especially
in scenarios characterized by a scarcity of labeled data.

2.2 SYNTHETIC DATA IN LLMS

Large language models (LLMs), a specialized subset of generative models tailored for the text do-
main, have demonstrated remarkable capabilities in generating high-quality text data. Similar to
traditional generative data augmentation, synthetic data produced by these models is increasingly
utilized to enhance LLMs, particularly during post-training phases. Given the scarcity of labeled
data in specific domains, synthetic data plays a crucial role in boosting the performance of LLMs
across a variety of downstream tasks, including text classification (Li et al., 2023), clinical text min-
ing (Tang et al., 2023), and code generation (Tsai et al., 2024). However, unlike classic generative
data augmentation, synthetic data within LLMs is typically generated by the language models them-
selves and often predominates the training data in post-training stages. This predominance stems
from the high-quality demands of synthetic data in LLM contexts, which necessitates alignment
with human intent. Efforts to enhance the quality of synthetic data in LLMs have included integrat-
ing methodologies such as active learning (Wagner et al., 2024) and reinforcement learning (Setlur
et al., 2024). Despite these advancements, the theoretical understanding of how synthetic data in-
fluences the learning process in LLMs remains limited. Key questions persist regarding the mecha-
nisms through which synthetic data impacts LLM training and the optimal strategies for designing
synthetic data to maximize LLM performance (Long et al., 2024). Addressing these questions is
essential for furthering our comprehension and utilization of synthetic data in enhancing large lan-
guage model efficacy.

2.3

INFORMATION BOTTLENECK THEORY & GENERALIZATION CAPABILITY

The information bottleneck (IB) theory, as introduced by (Tishby et al., 2000), serves as a theo-
retical construct designed to elucidate the learning processes within neural networks. In essence,
for a given Markov chain X → Z → Y , the IB theory aims to optimize the learning process by
maximizing the mutual information between Y and Z while minimizing the mutual information be-
tween X and Z (Hu et al., 2024). IB theory has been widely adopted across various deep learning
fields, such as text classification (Slonim et al., 2001), sentence summarization (West et al., 2019),

3

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

and image clustering (Hu et al., 2019). Expanding upon these foundations, further research has ex-
plored generalization error upper bounds that incorporate mutual information (Russo & Zou, 2019;
Xu & Raginsky, 2017). These studies have established a connection between the generalization
capabilities of deep neural networks (DNNs) and IB theory (Alquier et al., 2024). More recent ad-
vancements have also highlighted the links between mutual information bounds and the PAC-Bayes
framework (Banerjee & Mont´ufar, 2021). This type of bound suggests that the generalization error
is intrinsically limited by the relevance between the training data and the learned model parameters.

3 PRELIMINARIES

3.1 NOTATIONS & EXPERIMENTAL SETUP

Let Sanchor represent the real data utilized for generation, and Sgen denote the synthetically gener-
ated data. The LLM employed in the generation process is designated as M , with the input prompt
labeled as p. The distribution of the post-training target task T is referred to as D, while the output
distribution of the LLM is denoted by DM . Additionally, the distribution corresponding to the syn-
thetic data is represented as Dgen. The generalization error associated with the under-aligned LLM
π on the synthetic data Sgen is expressed as Err(πSgen ), and the generalization error related to the
anchor data is indicated by Err(πSanchor). We define H(·) as the entropy of a random variable, I(·, ·)
as the mutual information between two random variables, DKL as the Kullback-Leibler divergence,
and DTV as the total variation distance. The detailed definitions are listed in Appendix A.1.

To provide a more intuitive demonstration, we use an example in the Gaussian mixture model
(GMM) setting during the explanation.
In simple terms, we assume that the target of the post-
training task contains K + J Gaussian distribution components, and set up a corresponding ground-
truth GMM (gt-GMM, G) to represent the target of the post-training task. After that, we randomly
sample from the first K components of the gt-GMM as anchor data. To simulate the generative
model M , we added L random components to the gt-GMM, which may include extra distributions,
making M a GMM with total K + J + L components. Finally, we randomly sampled data from M
to obtain the simulated synthetic data. The detailed experimental setup is listed in Appendix B.

3.2 MODELING SYNTHETIC DATA GENERATION

Long et al. (2024) provided a brief summary for the synthetic data generation, the overall process
of synthetic data generation can be modeled as Sgen ← Mp (T , Sanchor ), where Sgen is the gener-
ated synthetic data, M is the generation model (usually a well-trained LLM), p is the prompt for
generation, T is the downstream task, and Sanchor is the anchor data (real data). More specifically,
the prompt p is derived from the generation task T and the anchor data Sanchor, and consists of three
crucial elements: p(T , Sanchor) ← E (etask, econdition, edemo), where E is the prompt template, etask
is the task element, econdition is the condition element, and edemo is the anchor data element. The
conceptual framework of this modeling is straightforward. Sgen essentially constitutes a modifica-
tion of the output generated by M in response to the prompt p, where the prompt p is defined by
the downstream task T and the anchor data Sanchor. The specifics of the generation process are thus
governed by the prompt p and M .

We enhance our understanding of synthetic data generation by reevaluating the distributional rela-
tionships among the anchor data Sanchor, the prompt p, and the synthetic data Sgen produced. We
postulate that the anchor data Sanchor is sampled from distribution D associated with the down-
stream task, and the generation process is influenced by both the prompt p and the generative
model M . Consequently, Sgen represents a modification of M ’s output in response to the prompt p:
Sgen = M (p) + ϵ, where ϵ is a noise term for the measurement of revision, such as the data curation
process.

The prompt p is intricately linked to the downstream task T and the anchor data Sanchor. We postulate
that Sanchor forms the core of the prompt p, upon which a task-specific transformation function ϕT
is applied. Consequently, the prompt p can be mathematically modeled as p = ϕT (Sanchor), where
ϕT is a function that maps the anchor data to the prompt, consists of all the task-relevant transfor-
mation, like the template and other customized settings for more faithful and diverse generation.

4

Under review as a conference paper at ICLR 2025

216

217
218
219
220
221

222
223
224
225
226
227

228
229
230
231
232
233

234
235
236
237
238
239

240
241
242
243
244

245
246
247
248
249
250

251
252
253
254
255
256

257
258
259
260
261
262

263
264
265
266
267

268
269

For simplicity, we note that Sanchor ∼ D, p ∼
Dp, M (p) ∼ DM (·|p), and Sgen ∼ Dgen,
and comprehensive details about the relation-
ships between the distributions are listed in Ap-
pendix C. The overall synthetic data generation
process in our modeling is depicted in Figure 1
(a). This illustration enhances our understand-
ing of the connection between the generation
process and distributions.

The lower part of Figure 1 (a) details the spe-
cific stages of data generation.
Initially, the
anchor data Sanchor undergoes a transformation
via the function ϕT to constitute the prompt p,
which in turn is used by the LLM M to gener-
ate the synthetic data Sgen, incorporating noise
ϵ. The upper portion of Figure 1 (a) delineates
the corresponding process of distribution shift.
Sanchor is derived from distribution D, and the
prompt p emerges from distribution Dp con-
ditioned on ϕT . The LLM M produces the
output M (p) from the conditional distribution
DM (·|p), and the final synthetic data Sgen is
sampled from Dgen, representing a convolution
of DM and Dϵ also conditioned on p.

Figure 2: The simulation of the distribution rela-
tionships with GMMs. “•” represents the anchor
data sampled from distributions colored blue, and
“•” represents the synthetic data sampled from
distributions colored orange.

Given that Dp relates solely to ϕT and D (or
Sanchor), and Dgen related only to M and p, the transition from Sanchor to p to Sgen, (i.e. Sanchor → p →
Sgen), constitutes a Markov chain. Figure 1 (b) provides a comprehensive view of the distributions
and the nature of the distribution shift discussed. Specifically, D is denoted as the orange circle,
and DM is denoted as the blue circle. After M being prompted on p, the conditioned distribution
DM (·|p) is denoted as all blue areas, and the final Dgen is represented as the deep blue area after the
compression on ϵ. This illustration aids in understanding that the generation process essentially
compresses the output distribution of M , DM , towards the post-training target distribution D,
based on the conditions imposed by the prompt p and noise ϵ.

To provide a clearer visualization, we simulate the distribution relationships using GMMs, the re-
sult is depicted in Figure 2. The distributions of Sgen are visualized as an effort to encompass the
distributions of Sanchor. However, since Sgen is derived from the model M , which incorporates more
complex distribution components, the distribution of Sgen not only attempts to mirror Sanchor but
also extends beyond, covering broader areas.

3.3 BRIDGING THE GENERALIZATION CAPABILITY

Subsection 3.2 offers an exhaustive examination of the synthetic data generation process, which
is pivotal for elucidating the generalization error associated with the under-aligned LLM π when
applied to synthetic data Sgen. This subsection endeavors to correlate the generalization error of π
on the synthetic data Sgen with the synthetic data generation process as previously delineated.

Given the focus on the alignment task performance, and considering that π is a pre-trained LLM,
then is subsequently trained on synthetic data sampled from Dgen, the generalization error of post-
(cid:12)
(cid:12)
(cid:12)
(cid:12)
trained LLM πSgen is delineated as Err(πSgen ) =
(cid:12)RD(πSgen) − (cid:98)RSgen(πSgen)
(cid:12), where D is the real
(cid:2)ℓ(πSgen , z)(cid:3) denotes the true error of πSgen
distribution of the post-training task. RD(πSgen) = Ez∼D
(cid:2)ℓ(πSgen, z)(cid:3) denotes the empirical error of πSgen
on the distribution D, and (cid:98)RSgen (πSgen) = 1
on the synthetic data. Similar like Zheng et al. (2023b), and by the definition of the synthetic data
generation process, we can simplify the above upper bound as the following lemma:

n Σz∈Sgen

Lemma 3.1. Assume that π is with a loss function ℓ bounded by C, given an i.i.d. synthetic dataset
Sgen generated as the above defined, then the following synthetic data training generalization error

5

Dimension1Dimension2Under review as a conference paper at ICLR 2025

Figure 3: Illustration about the reverse bottleneck effect and comparison with classic ML process.
Left: the similarity between the forward process of synthetic data generation and classic ML. Right:
the difference between the information flow of the two process, where synthetic data generation
gains information from M , constituting a reverse-bottleneck.

upper bound holds:

Err(πSgen) ≤ C (DTV(D, DM ) + DTV(DM , Dgen))
(cid:125)

(cid:123)(cid:122)
Distributions’ Divergence

(cid:124)

+

(cid:12)
(cid:12)
(cid:12)RDgen(πSgen ) − (cid:98)RSgen (πSgen )
(cid:12)
(cid:12)
(cid:12)
(cid:125)
(cid:123)(cid:122)
(cid:124)
Generalization Error w.r.t. synthetic data

.

(1)

The proof is referred to the Appendix D. The divergences can be defined as the task divergence
(DTV(D, DM )) and the generation divergence (DTV(DM , Dgen)), which is denoted in Figure 1 (b).
The task divergence is determined by the ability of the LLM M and the relevance with the task T .
The generation divergence is determined by the generation process including the prompt engineering
and the data curation. In the training practice, the two divergences are controlled by either the strong
ability of M or the strict prompt engineering, this partially explains why synthetic data is effective.

4 MAIN RESULT

In this section, we delves deeper into the implications of the synthetic data generation process on
the generalization capabilities.

4.1

INFORMATION GAIN & REVERSE-BOTTLENECK

To enhance our understanding of the synthetic data generation process, we delineate a suite of con-
cepts pertaining to the information-flow within this process. Initially, we introduce the notion of
synthetic factors, which represent the fundamental elements that influence the formation of Sgen.
Definition 4.1. (Synthetic factors.) Assume that the synthetic data Sgen = M (p) + ϵ is derived from
two factors, i.e. M (p) = h(ep) + g(eM ). The ep represents the factor w.r.t. prompt p and the eM
represents the factor w.r.t. applied LLM M .

With the synthetic factors established, we posit that the synthetic data Sgen is primarily governed by
two distinct factors: ep and eM , which are actually assumed random variables related to the prompt
p and the LLM M respectively. Following this framework, we proceed to introduce the concept of
information gain within the context of the synthetic data generation process.
Definition 4.2. (Information gain.) The information gain in the synthetic data generation process
is defined as:

∆I = H(M (p)) − I (h(ep), M (p)) .

(2)

The information gain, denoted as ∆I, serves as a metric for assessing the enhancement of informa-
tion in the synthetic data generation process. It quantifies the incremental information content from
the prompt p to the synthetic data Sgen, specifically, the information introduced by the LLM M .

6

270

271
272
273
274
275

276
277
278
279
280
281

282
283
284
285
286
287

288
289
290
291
292
293

294
295
296
297
298

299
300
301
302
303
304

305
306
307
308
309
310

311
312
313
314
315
316

317
318
319
320
321

322
323

S!"#$%&𝑝S’("𝜙𝒯𝑀𝑋𝑍𝑌𝑒𝑛𝑐𝑑𝑒𝑐Information Flow(latent)𝑿𝒁𝒀bottleneck𝑺𝒂𝒏𝒄𝒉𝒐𝒓𝒑𝑴reverse-bottleneck(latent)Forward ProcessSynthetic DataGenerationClassic MLUnder review as a conference paper at ICLR 2025

324

325
326
327
328
329

330
331
332
333
334
335

336
337
338
339
340
341

342
343
344
345
346
347

348
349
350
351
352

353
354
355
356
357
358

359
360
361
362
363
364

365
366
367
368
369
370

371
372
373
374
375

376
377

In alignment with the classical information bottleneck theory, we also introduce the concept of a
compression bottleneck, which is defined in the context of synthetic factors.
Definition 4.3. (Compression bottleneck.) We consider the compression bottleneck of the synthetic
data towards the post-trained model parameter W as:

Bsyn = I(eM , W ) + I(ep, W ).

(3)

Having delineated the concepts of information gain and compression bottleneck, we now advance
our discussion to clarify the information flow within the synthetic data generation process, introduc-
ing the notion of a reverse-bottleneck effect. This framework acknowledges that the distribution Dp
is directly influenced by ϕT and Danchor (or Sanchor), while Dgen pertains solely to M and p. Conse-
quently, the sequence Sanchor → p → ep → W constitutes a Markov chain. Similarly, the process
M (p) → eM → W also forms a Markov chain.

The former Markov chain, as depicted in the left part of Figure 3, parallels a classical machine
learning (ML) process, in which the input X is transformed into a latent representation Z via an
encoder, and then Z is further decoded into the output Y through a decoder. Similarly, in the
synthetic data generation process, the input Sanchor is converted to p (which is often assumed as latent
in practical applications) via ϕT , and subsequently p is transformed into Sgen by M . However, the
presence of the latter Markov chain introduces a crucial distinction between the two processes from
an information flow perspective due to the prior knowledge embedded by M . As illustrated in the
right part of Figure 3, unlike classic ML process, the synthetic data generation process leverages
M to facilitate information gains, thereby enriching the informational content of Sgen.

This perspective emphasizes the distinctive dynamics and augmented capabilities of the synthetic
data generation process in terms of capturing and utilizing information. Subsequently, we aim to
analyze the relationship between the information gain and the generalization error of the model after
training on the synthetic data.

4.2

INFORMATION-FLOW GENERALIZATION ERROR UPPER BOUND

In this subsection, we endeavor to derive the upper bounds of the generalization error from an
information-flow perspective, employing the concepts previously defined. We initiate our analy-
sis with a classical information upper bound applicable to deep neural networks, as elaborated in
Lemma 4.4 (Zhang et al., 2018).
Lemma 4.4. For a deep neural network with L hidden layers, input S, and parameters W . The loss
function is σ-sub-Gaussian with respect to (W, Z) given any w, if all L hidden layers are contraction
layers, the expected generalization error can be bounded as follows,

E [R(W ) − RS(W )] ≤ exp

(cid:18)

−

L
2

log

1
η

(cid:19) (cid:114)

2σ2
n

I(S, W ).

(4)

Lemma 4.4 establishes a connection between the expected generalization error and the mutual in-
formation between training data S and learned model parameters W . Despite network depth L and
instance volume n, the principal constraint is imposed by the mutual information term.

Accordingly, in scenarios where post-training is with synthetic data, the generalization error is in-
herently constrained by the mutual information between the synthetic data Sgen and LLM parameters
after training, denoted as I(Sgen, W ). Characterizing this term presents a significant challenge due
to the difficulty in measuring mutual information accurately. To address this, we introduce an ana-
lytical upper bound for I(Sgen, W ) in Lemma 4.5 to facilitate a more comprehensive understanding
of the dynamics influencing model performance in post-training.
Lemma 4.5. (Information-flow upper bound.) Given a synthetic dataset Sgen defined above, and
model parameters W learned from Sgen, the mutual information term I(Sgen, W ) can be bounded
by the following inequality:

I(Sgen, W ) ≤ −∆I + Bsyn + H(eM ) + δϵ,p,

(5)

where δϵ,p indicates the efficiency during the data curation and model prompting process, which
is detailed in the proof in Appendix E. Together with Lemma 4.4, we can further derive an upper
bound for a training procedure with relation to the synthetic data defined above in Lemma 4.6.

7

Under review as a conference paper at ICLR 2025

Lemma 4.6. (Generalization error upper bound w.r.t. synthetic data.) For a deep neural network π
with L hidden layers, the parameters W are optimized from synthetic data Sgen described aboved.
The loss function is σ-sub-Gaussian with respect to (W, Z) given any w, if all L hidden layers are
contraction layers, the expected generalization error can be bounded as follows:

E

(cid:12)
(cid:12)RDgen (πSgen ) − (cid:98)RSgen (πSgen)
(cid:12)

(cid:12)
(cid:12)
(cid:12) ≤ exp

(cid:19) (cid:114)

(cid:18)

−

L
2

log

1
η

2σ2 [−∆I + Bsyn + H(eM ) + δϵ,p]
n

.

(6)

Lemma 4.6 delineates a quantifiable upper bound for the expected generalization error in relation to
synthetic data. Beyond basic configuration parameters such as network depth L and data size n, this
upper bound is determined by four key factors outlined in the corresponding remarks.

Remark 1. ∆I quantifies the information gain during the data generation process. This bound
demonstrates that an increase in information extracted from the model M enhances the quality of
the generated data.

Remark 2. Bsyn denotes the compression bottleneck, which is defined as the mutual information
between synthetic factors and the model parameters W . A more pronounced compression of this
term leads to improved generalization performance.

Remark 3. H(eM ) represents the entropy associated with the synthetic factor relative to the model
M . Intuitively, reducing this entropy by choosing a model M more aligned with the specific tasks
can substantially enhance downstream generalization.

Remark 4. δϵ,p concerns the efficiency during the data curation and model prompting process,
highlighting the impact of noise and other data degradation factors on the overall data utility.

These factors collectively influence the generalization performance, indicating that a better general-
ization ability can be achieved by enhancing the information gain, reducing the compression bottle-
neck, minimizing the entropy, and balancing the efficiency. Finally, by integrating the insights from
Lemma 3.1, the overall upper bound of the expected generalization error in the LLM post-training
with synthetic data can be derived as a comprehensive boundary in Theorem 4.7.

Theorem 4.7. (Synthetic data post-training upper bound.) For the same condition as lemma 4.6
and a synthetic data generation process described above, the generalization error of the model π
post-trained on the synthetic data can be bounded as:

E(Err(πSgen)) ≤ C (DTV(D, DM ) + DTV(DM , Dgen))
(cid:125)

(cid:124)

(cid:123)(cid:122)
Distributions’ Divergence
(cid:19) (cid:114)

(cid:18)

+ exp

−

L
2

(cid:124)

log

1
η

2σ2 [−∆I + Bsyn + H(eM ) + δϵ,p]
n

(7)

.

(cid:123)(cid:122)
Generalization Error w.r.t. synthetic data

(cid:125)

4.3 GENERALIZATION GAIN WITH SYNTHETIC DATA

Theorem 4.7 establishes a general upper bound for the generalization error of LLMs post-trained
with synthetic data. In this section, our objective is to analyze the generalization gains achieved by
using synthetic data compared to scenarios devoid of synthetic data.
We commence our analysis with the anchor data Sanchor. Analogous to the definition of Err(πSgen),
the generalization error of an LLM that has been post-trained on Sanchor is defined as Err(πSanchor ) =
(cid:12)
(cid:12)
(cid:12)
(cid:12)
(cid:12)RD(πSanchor) − (cid:98)RSanchor(πSanchor)
(cid:12). It is logically sound to assume that Sanchor is sampled from the
distribution D. Building upon Lemma 4.4 and assume that Sanchor comprises m instances, we can
derive the subsequent result in Lemma 4.8.

Lemma 4.8. (Anchor data post-training upper bound.) For the same condition as lemma 4.6, the
generalization error of the model π post-trained on the anchor data can be bounded as:

E(Err(πSanchor )) ≤ exp

(cid:18)

−

L
2

log

1
η′

(cid:19) (cid:114)

2σ2
m

I(Sanchor, W ′),

(8)

8

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Under review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

′

′

are the variables of the model trained with Sanchor, noted different from that of
where η
and W
model trained with Sgen. η
is the model pa-
rameters. Given that m << n typically applies in real-world scenarios, Lemma 4.8 often represents
a less stringent upper bound compared to Lemma 4.4, this results in potentially poorer generalization
when relying solely on Sanchor rather than utilizing synthetic data.

is a constant depending on the information loss and W

′

′

But a pertinent question arises: do other aspects of synthetic data generation, beyond the influ-
ence of data size, also contribute to improvements in generalization performance? Our focus is
on examining how various elements within the synthetic data process impact generalization during
post-training. It is inappropriate, however, to directly compare other components across these two
bounds due to variations in loss and training data specifics, which affect the parameters η and W
differently, where η represents a measure of information compression and is challenging to quan-
tify accurately (Zhang et al., 2018). Thus, our analysis primarily centers on the mutual information
) and I(Sgen, W ). To systematically evaluate the generalization capabilities con-
terms I(Sanchor, W
ferred by synthetic data in relation to these mutual information metrics, we introduce a definition for
generalization gain measurement as Definition 4.9.
Definition 4.9. (Generalization Gain via Mutual Information, GGMI.) GGMI is defined as the dif-
ference between the mutual information terms in the two generalization upper bounds:

′

GGMI = I(Sanchor, W

′

) − I(Sgen, W ).

(9)

′

A larger upper bound for the GGMI signifies greater potential generalization benefits when utilizing
synthetic data. To elucidate the impact of synthetic data on model generalization, we isolate the
influence of W
and establish that the GGMI can be effectively bounded.
Theorem 4.10. (Upper bound of GGMI.) Given the synthetic data generation above, W
is param-
eterized by training with Sanchor, and W is parameterized by training with Sgen, the GGMI can be
bounded as follows:

′

GGMI ≤ ∆I − (α + 1)H(Sanchor|W ) + 2∆H + H(Sgen|W ) + ϵW,p,
(10)
where ∆H = H (Sanchor) − H (Sgen), ϵW,p = H(Sanchor|W ) − H(Sanchor|M (p)), it is assumed that
H(Sanchor|W

) = αH(Sanchor|W ), α ≥ 0.

′

The proof is referred to Appendix F. Consequently, we proceed to conduct a thorough analysis of
each component specified in Theorem 4.10.

Remark 1. ∆I represents the information gain derived from the model M . An increase in this
information gain typically leads to improved generalization capability for πSgen compared to πSanchor,
as the model leverages additional insights to enhance performance.

Remark 2. H(Sanchor|W ) indicates the conditional entropy between the anchor data Sanchor and the
model parameters W . For a larger upper bound of GGMI, it is encouraged to decrease this value by
strengthen the relevance between model parameters W and anchor data Sanchor.

Remark 3. ∆H denotes the entropy decrease when generating synthetic data Sgen from anchor data
Sanchor. It implies that more uncertainty is eliminated during synthetic data generation leads to more
generalization ability.

Remark 4. H(Sgen|W ) reflects the conditional entropy between the synthetic data Sgen and the
model parameters W . Weakening the relevance between these two entities is encouraged to ensure
that the model learns the general pattern of synthetic data thus leading to better generalization.

Remark 5. ϵW,p denotes the effect of information compression by the training algorithm. A more
pronounced compression effect typically results in a higher value, suggesting that efficient data
representation contributes positively to model efficacy.

As emphasized in (Long et al., 2024), the generation of synthetic data typically focuses on two
primary objectives: faithfulness and diversity. These objectives are associated with ∆H and ∆I,
respectively. Specifically, ∆H, which quantifies the entropy decrease during synthetic data genera-
tion, as presented in Theorem 4.10, encourages the model to eliminate uncertainty during synthetic
data generation, thereby enhancing the faithfulness of the synthetic data. In addition, ∆I serves as
a measurement of the additional information introduced by the generative model M . Given that M
is typically pre-trained on a more extensive dataset, ∆I in Theorem 4.10 promotes the objective of
diversity by facilitating greater information gain from M .

9

Under review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Figure 4: KL Gap with different components settings. By default, we set K = J = L = 2, and vary
each of them from 2 to 15 to observe the corresponding change of KL Gap. An increase of KL Gap
is observed when J increases, while a decrease is observed when K and L increase. The shading
indicates the standard deviation of 100 rounds of random settings.

4.4 VERIFICATION WITH GMM SIMULATION

Building upon the simulation settings, we offer a straightforward validation of the theoretical results
discussed above. Specifically, we first fit a GMM π comprising K +J +L components to both Sanchor
and Sgen, yielding πSanchor and πSgen respectively. We then introduce a metric termed KL Gap, defined
as DKL(πSanchor||G) − DKL(πSgen||G), which represents the difference of KL-divergence between
the fitted GMMs (πSanchor and πSgen) and the ground-truth GMM G. A larger KL Gap corresponds to
a greater GGMI, indicating enhanced generalization benefits from synthetic data.

To control the variables outlined in Theorem 4.10, we adjust the number of components in the
GMM M and the ground-truth GMM G. The result is illustrated in Figure 4. Generally, increasing
J facilitates the scaling of ∆I, resulting in a larger upper bound for GGMI. In contrast, larger K
amplifies the influence of anchor data within the post-training target distribution, thereby increasing
the H(Sanchor|W ) term and tightening the upper bound of GGMI. Additionally, while an increase
in L enhances H(Sgen|W ), it concurrently leads to a reduction in ∆H. As a result, we observe a
trade-off manifested as a decrease in the KL Gap in our simulation outcomes.

5 CONCLUSION

In this paper, we have conducted a detailed analysis of synthetic data utilization in post-training
large language models (LLMs). We present a comprehensive modeling of the current synthetic data
generation process, focusing on its distributional aspects, which further connects the generalization
capabilities of post-trained models. We introduce a novel reverse-bottleneck framework, allowing
us to derive a measurable upper bound on generalization errors. Our analysis reveals that the pivotal
constraint on generalization ability is influenced by the information gain from the generative model
M . Additionally, we present the Generalization Gain via Mutual Information (GGMI), showing that
larger information gains enhance the generalization capability of post-trained models. We empha-
size the importance of balancing faithfulness and diversity during post-training stages, providing
a theoretical foundation for existing methodologies. Unfortunately, due to limitations in computa-
tional resources, we are unable to validate our findings within real-world LLM settings. Looking
ahead, future research should focus on developing adaptive models that respond to the evolving char-
acteristics of synthetic data. This includes enhancing generative models and fine-tuning parameters
for specific learning scenarios, as well as exploring various generative models to better replicate
real-world data complexities while improving model performance.

10

23456789101112131415K42024KL GapKL Gap vs K23456789101112131415J202468KL GapKL Gap vs J23456789101112131415L420246KL GapKL Gap vs LUnder review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, and et al. The falcon series of open

language models, 2023. URL https://arxiv.org/abs/2311.16867.

Pierre Alquier et al. User-friendly introduction to pac-bayes bounds. Foundations and Trends® in

Machine Learning, 17(2):174–303, 2024.

Jinze Bai, Shuai Bai, Yunfei Chu, and et al. Qwen technical report, 2023. URL https://arxi

v.org/abs/2309.16609.

Pradeep Kr Banerjee and Guido Mont´ufar. Information complexity and generalization bounds. In
2021 IEEE International Symposium on Information Theory (ISIT), pp. 676–681. IEEE, 2021.

Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick
Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open
instruction-tuned llm, 2023. URL https://www.databricks.com/blog/2023/04/
12/dolly-first-open-commercially-viable-instruction-tuned-llm.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and et al. The llama 3 herd of models, 2024.

URL https://arxiv.org/abs/2407.21783.

Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large

language models using semantic entropy. Nature, 630(8017):625–630, 2024.

Oscar Giles, Kasra Hosseini, Grigorios Mingas, Oliver Strickson, Louise Bowler, Camila Rangel
Smith, Harrison Wilde, Jen Ning Lim, Bilal Mateen, Kasun Amarasinghe, et al. Faking feature
importance: A cautionary tale on the use of differentially-private synthetic data. arXiv preprint
arXiv:2203.01363, 2022.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair,
Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information
processing systems, 27, 2014.

Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Sch¨olkopf. Measuring statistical de-
pendence with hilbert-schmidt norms. In International conference on algorithmic learning theory,
pp. 63–77. Springer, 2005.

Shizhe Hu, Xiaoqiang Yan, and Yangdong Ye. Multi-task image clustering through correlation
propagation. IEEE Transactions on Knowledge and Data Engineering, 33(3):1113–1127, 2019.

Shizhe Hu, Zhengzheng Lou, Xiaoqiang Yan, and Yangdong Ye. A survey on information bottle-

neck. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.

Xuming Hu, Aiwei Liu, Zeqi Tan, Xin Zhang, Chenwei Zhang, Irwin King, and Philip S Yu.
Gda: Generative data augmentation techniques for relation extraction tasks. arXiv preprint
arXiv:2305.16663, 2023.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child,
Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language
models. arXiv preprint arXiv:2001.08361, 2020.

Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.

Seonmin Koo, Chanjun Park, Seolhwa Lee, Jaehyung Seo, Sugyeong Eo, Hyeonseok Moon, and
Heuiseok Lim. Uncovering the risks and drawbacks associated with the use of synthetic data for
grammatical error correction. IEEE Access, 2023.

Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. Synthetic data generation with large lan-
guage models for text classification: Potential and limitations. arXiv preprint arXiv:2310.07849,
2023.

Hao Liang, Linzhuang Sun, Jingxuan Wei, Xijie Huang, Linkun Sun, Bihui Yu, Conghui He, and
Wentao Zhang. Synth-empathy: Towards high-quality synthetic empathy data. arXiv preprint
arXiv:2407.21669, 2024.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On
llms-driven synthetic data generation, curation, and evaluation: A survey, 2024. URL https:
//arxiv.org/abs/2406.15126.

Wan-Duo Kurt Ma, JP Lewis, and W Bastiaan Kleijn. The hsic bottleneck: Deep learning without
back-propagation. In Proceedings of the AAAI conference on artificial intelligence, volume 34,
pp. 5085–5092, 2020.

Adyasha Maharana and Mohit Bansal. Grada: Graph generative data augmentation for common-
sense reasoning. In Proceedings of the 29th International Conference on Computational Linguis-
tics, pp. 4499–4516, 2022.

Meta. Introducing llama 3.2, 2024. URL https://www.llama.com/docs/model-cards

-and-prompt-formats/llama3_2.

Anders Giovanni Møller, Jacob Aarup Dalsgaard, Arianna Pera, and Luca Maria Aiello. The
parrot dilemma: Human-labeled vs. llm-augmented data in classification tasks. arXiv preprint
arXiv:2304.13861, 2023.

OpenAI. Gpt-4o system card, 2024. URL https://arxiv.org/abs/2410.21276.

OpenAI, Josh Achiam, Steven Adler, and et al. Gpt-4 technical report, 2024. URL https:

//arxiv.org/abs/2303.08774.

Jeiyoon Park, Chanjun Park, and Heuiseok Lim. Chatlang-8: An llm-based synthetic data generation

framework for grammatical error correction. arXiv preprint arXiv:2406.03202, 2024.

Ajay Patel, Colin Raffel, and Chris Callison-Burch. Datadreamer: A tool for synthetic data genera-

tion and reproducible llm workflows. arXiv preprint arXiv:2402.10379, 2024.

Chen Qian, Jie Zhang, Wei Yao, Dongrui Liu, Zhenfei Yin, Yu Qiao, Yong Liu, and Jing Shao.
Towards tracing trustworthiness dynamics: Revisiting pre-training period of large language mod-
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association
els.
for Computational Linguistics: ACL 2024, pp. 4864–4888, Bangkok, Thailand, August 2024.
Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.290. URL
https://aclanthology.org/2024.findings-acl.290.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language

models are unsupervised multitask learners. 2019.

Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Interna-

tional conference on machine learning, pp. 1530–1538. PMLR, 2015.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-
resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer-
ence on computer vision and pattern recognition, pp. 10684–10695, 2022.

Daniel Russo and James Zou. How much does your data exploration overfit? controlling bias via

information usage. IEEE Transactions on Information Theory, 66(1):302–323, 2019.

Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. Rl on
incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold. arXiv preprint
arXiv:2406.14532, 2024.

Noam Slonim, Naftali Tishby, et al. The power of word clusters for text classification.
European Colloquium on Information Retrieval Research, volume 1, pp. 200, 2001.

In 23rd

Ruixiang Tang, Xiaotian Han, Xiaoqian Jiang, and Xia Hu. Does synthetic data generation of llms

help clinical text mining? arXiv preprint arXiv:2303.04360, 2023.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy
Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model.
https://github.com/tatsu-lab/stanford_alpaca, 2023.

12

Under review as a conference paper at ICLR 2025

Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv

preprint physics/0004057, 2000.

Yun-Da Tsai, Mingjie Liu, and Haoxing Ren. Code less, align more: Efficient llm fine-tuning for

code generation with data pruning. arXiv preprint arXiv:2407.05040, 2024.

Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and An Chang

Ho. Will we run out of data? limits of llm scaling based on human-generated data. 2022.

Stefan Sylvius Wagner, Maike Behrendt, Marc Ziegele, and Stefan Harmeling. Sqbc: Active learn-
ing using llm-generated synthetic data for stance detection in online political discussions. arXiv
preprint arXiv:2404.08078, 2024.

Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii
Kuchaiev, and Yi Dong. Helpsteer2-preference: Complementing ratings with preferences, 2024.
URL https://arxiv.org/abs/2410.01257.

Peter West, Ari Holtzman, Jan Buys, and Yejin Choi. Bottlesum: Unsupervised and self-
supervised sentence summarization using the information bottleneck principle. arXiv preprint
arXiv:1909.07405, 2019.

Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learn-

ing algorithms. Advances in neural information processing systems, 30, 2017.

Shin’ya Yamaguchi, Sekitoshi Kanai, and Takeharu Eda. Effective data augmentation with multi-
In Proceedings of the AAAI Conference on Artificial Intelligence, vol-

domain learning gans.
ume 34, pp. 6566–6574, 2020.

Yuwei Yin, Jean Kaddour, Xiang Zhang, Yixin Nie, Zhenguang Liu, Lingpeng Kong, and Qi Liu.
Ttida: Controllable generative data augmentation via text-to-text and text-to-image models, 2023.
URL https://arxiv.org/abs/2304.08821.

Jingwei Zhang, Tongliang Liu, and Dacheng Tao. An information-theoretic view for deep learning.

arXiv preprint arXiv:1804.09060, 2018.

Chenyu Zheng, Guoqiang Wu, and Chongxuan Li. Toward understanding generative data augmen-

tation. Advances in neural information processing systems, 36:54046–54060, 2023a.

Chenyu Zheng, Guoqiang Wu, and Chongxuan Li. Toward understanding generative data augmen-

tation, 2023b. URL https://arxiv.org/abs/2305.17476.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang,
Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and
chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023c.

13

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Under review as a conference paper at ICLR 2025

A DEFINITION AND INTRODUCTION ABOUT INFOMATION BOTTLENECK

THEORY

A.1 DEFINITION OF NOTATIONS

We summarize the notations used in subsection 3.1 and provide their specific definitions.

First, we define the notations about the concept related to information entropy.
Definition A.1. (Entropy of a random variable.) The entropy of a random variable X is defined as:
(cid:88)

H(X) = −

p(x) log p(x),

For continual random variable, the entropy is defined as:

x

(cid:90)

H(X) = −

p(x) log p(x)dx.

The entropy is a measurement of the uncertainty of the random variable, and the larger the entropy,
the more uncertain the random variable is. It can also be considered as the average information
content of the random variable.
Definition A.2. (Conditional entropy of a random variable.) The conditional entropy of a random
variable X given another random variable Y is defined as:

H(X|Y ) = −

(cid:88)

x,y

p(x, y) log p(x|y).

For continual random variable, the conditional entropy is defined as:

H(X|Y ) = −

(cid:90)

p(x, y) log p(x|y)dxdy.

The conditional entropy is a measurement of the uncertainty of the random variable X given the
information of the random variable Y . It can also be considered as the average information content
of the random variable X with Y given.

Building upon the definitions above, we can further define the concepts we used in the main text
with relation to information theory, including relative entropy, total variation distance, and mutual
information.
(Relative entropy or Kullback-Leibler divergence.)
Definition A.3.
Kullback-Leibler divergence between two probability distributions p and q is defined as:

The relative entropy or

DKL(p∥q) =

p(x) log

p(x)
q(x)

.

(cid:88)

x

The relative entropy serves as a measurement of the difference between two probability distributions.
Definition A.4. (Total variation distance.) The total variation distance between two probability
distributions p and q on a finite or countable set E is defined as:

DTV(p, q) = supA∈E |p(A) − q(A)|
(cid:88)

|p(x) − q(x)| .

=

1
2

x∈E

The total variation distance is also a measurement of the difference between two probability distri-
butions.
Definition A.5. (Mutual information.) The mutual information between two random variables X
and Y is defined as:

I(X, Y ) = H(X) − H(X|Y ).

The mutual information is a measurement of the amount of information that one random variable
contains about another random variable. The larger the mutual information, the more information
the two random variables share.

14

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Under review as a conference paper at ICLR 2025

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Figure 5: Illustration of the setup of the GMMs for simulation.

A.2 THE INFORMATION BOTTLENECK THEORY

The information bottleneck (IB) theory is a theoretical construct designed to elucidate the learning
processes within neural networks. In essence, for a given Markov chain X → Z → Y , the IB
theory aims to optimize the learning process by maximizing the mutual information between Y and
Z while minimizing the mutual information between X and Z. The optimization objective in IB
theory is generally expressed as:

L [p (z|x)] = I(Z, X) − βI(Z, Y ).

(11)

Originally developed within the context of information compression, IB theory has been widely
adopted across various deep learning fields, and further research has explored generalization error
upper bounds that incorporate mutual information (Russo & Zou, 2019; Xu & Raginsky, 2017).
These studies have established a connection between the generalization capabilities of deep neural
networks (DNNs) and IB theory (Alquier et al., 2024). A representative formulation of a general-
ization error upper bound from the perspective of mutual information is as follows:

genErr ≤

(cid:114)

2σ2
n

I(S, W ),

(12)

where S and W are training data and model parameters respectively, with the assumption that the
loss is σ-subgaussian. This type of bound suggests that the generalization error is intrinsically
limited by the relevance between the training data and the learned model parameters.

B DETAILS OF EXPERIMENTAL SETTINGS

We utilize Gaussian Mixture Models (GMMs) to simulate the data generation process, as illustrated
in Figure 5. Overall, we use a gt-GMM to simulate the ground-truth, or the post-training target
distribution, and a GMM M to simulate the generative model applied in the data generation process.

Their are three parts of components in the GMMs: the anchor sample part with K components, the
unsampled part with J components, and task-irrelevant part with L components. It is assumed that
the post-training target distribution is a combination of the anchor sample part and the unsampled
part, thus the gt-GMM contains K + J components from the anchor sample part and the unsampled
part, which is denoted as blue in Figure 5. However, the anchor data is only sampled from the anchor
sample part. This is a reasonable assumption for the real-world scenario, since the anchor data is
sparse and hard to cover the whole post-training task distribution.

Additionally, the generative model M is assumed to be a GMM with K +J +L components. Except
for the post-training target distribution, M also contains a task-irrelevant part, which is denoted as
orange in Figure 5. This is due to the fact that the generative model is always pre-trained on a larger
scale of data, and may not be perfectly aligned with the post-training target distribution, and may
introduce task-irrelevant components in the synthetic data generation process.

15

𝐾𝐽𝐿gt-GMMMsamplinganchor sample partunsampledparttask-irrelevantpartAnchor Data𝑆!"#$%&samplingSyntheticData𝑺𝒈𝒆𝒏Under review as a conference paper at ICLR 2025

Building upon the settings above, we sample from the anchor sample part components of the gt-
GMM to generate the anchor data Sanchor, and sample from the generative model M to generate
the synthetic data Sgen.
In the experiment, we set the dimension of the data to be d = 2, and
K = J = L = 2 by default, to facilitate the visualization and analysis of the data generation
process.

For simulation in the main text, we set the number of initial anhcor data N = 50 for each anchor
sample part component, and resample the 1000 data points for both GMM fitted on Sanchor and Sgen.
For the simulation to evaluate the KL Gap, the results are averaged over the 100 rounds, where for
each round, we also resample the final data points for 100 rounds.

C DETAILS OF SYNTHETIC DATA GENERATION MODELING

In this section, we elaborate on the modeling aspects of synthetic data generation, particularly fo-
cusing on the distributions of the prompt p and synthetic data Sgen, which are central to the process
of generating synthetic data for training large language models (LLMs).

Distribution of p: The prompt p is is derived from the transformation function ϕT , applied to the
anchor data Sanchor. This function is assumed to be reversible, allowing us to explore its properties
in the context of data generation:

p = ϕT (Sanchor),
where ϕT integrates various task-specific and conditional elements, defined as etask and econdition.
Assuming that ϕT is reversible, we can derive the distribution of p through the probability density
function (PDF) of Danchor (denoted as fDanchor ), the distribution of p can be modeled as follows:

p ∼ Dp(ϕT ) = Dϕ−1

T

,

where the PDF of Dϕ−1

T

is expressed as:

fϕ−1

T

(x) = fDanchor(ϕ−1

T (x))

(cid:12)
(cid:12)
(cid:12)
(cid:12)

det

(cid:18) ∂ϕ−1
T
∂x

(cid:19)(cid:12)
(cid:12)
(cid:12)
(cid:12)

,

which indicates how changes in Danchor influence the distribution of p through the transformation
function, taking into account the Jacobian determinant of the inverse transformation.

Distribution of Sgen: The synthetic data Sgen is the output of the large language model M when
prompted with p, typically augmented with noise ϵ to introduce variability and improve robustness.
Assuming that the output of M follows a specific distribution DM , based on the conditioning on p,
we represent the distribution of M (p) as:

M (p) ∼ DM (· | p),
The distribution of Sgen then combines the model’s output with noise, which is mathematically
characterized by the convolution of DM (·|p) and Dϵ:

Sgen ∼ Dgen(M, p) = DM (·|p) ∗ Dϵ,
where ∗ is the convolution operator, integrating the noise distribution Dϵ into the output distribution
of the model. This convolution reflects how noise impacts the precision and variability of the gener-
ated synthetic data, thus affecting the overall utility and effectiveness of the synthetic data in model
training.

Through these detailed formulations, we aim to provide a more granular understanding of how syn-
thetic data is modeled and generated, facilitating better integration and utilization in LLM training
processes. This deeper insight into the synthetic data generation mechanics enables more targeted
and effective training strategies, optimizing the performance of large language models in diverse
applications.

D PROOF OF LEMMA 3.1

Proof. Similar like Zheng et al. (2023b), we can further decompose the generalization error into the
following three components:
(cid:12)RD(πSgen) − RDM (πSgen)(cid:12)
Err(πSgen) ≤ (cid:12)
(cid:12) + (cid:12)
(cid:12)
(cid:12)
(cid:12)RDgen (πSgen) − (cid:98)RSgen (πSgen)
(cid:12)
(cid:12)
(cid:12) .

(cid:12)RDM (πSgen) − RDgen(πSgen )(cid:12)
(cid:12)

(13)

+

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

For the first item in lemma, we have:

(cid:12)RD(πSgen) − RDM (πSgen)(cid:12)
(cid:12)

(cid:12) =

≤

(cid:90)

(cid:12)
(cid:12)
(cid:12)
(cid:12)
(cid:90)

ℓ(πSgen, z) (PD(z) − PDM (z)) dz

z
(cid:12)ℓ(πSgen, z) (PD(z) − PDM (z))(cid:12)
(cid:12)
z
(cid:90)

(cid:12) dz

(cid:12)
(cid:12)
(cid:12)
(cid:12)

≤ C

|PD(z) − PDM (z)|

z

Similarly, for the second item in lemma, we have:

≲ CDTV(D, DM ).

(cid:12)RDM (πSgen) − RDgen(πSgen)(cid:12)
(cid:12)

(cid:12) =

≤

(cid:90)

(cid:12)
(cid:12)
(cid:12)
(cid:12)
(cid:90)

(cid:12)
(cid:12)
(cid:12)
(cid:12)

ℓ(πSgen, z) (cid:0)PDM (z) − PDgen(z)(cid:1) dz

z
(cid:12)
(cid:12)ℓ(πSgen, z) (cid:0)PDM (z) − PDgen(z)(cid:1)(cid:12)
z
(cid:90)

(cid:12)PDM (z) − PDgen(z)(cid:12)
(cid:12)
(cid:12)

(cid:12) dz

≤ C

z

Together with Eq. (13), Eq. (14), and Eq. (15), we have:

≲ CDTV(DM , Dgen).

(14)

(15)

(cid:12)RD(πSgen) − RDM (πSgen)(cid:12)
Err(πSgen ) ≤ (cid:12)
(cid:12) + (cid:12)
(cid:12)
(cid:12)
(cid:12)RDgen(πSgen) − (cid:98)RSgen(πSgen)
(cid:12)
(cid:12)
(cid:12)

+

(cid:12)RDM (πSgen) − RDgen (πSgen)(cid:12)
(cid:12)

(cid:12)
(cid:12)
(cid:12)RDgen(πSgen) − (cid:98)RSgen(πSgen)
(cid:12)
(cid:12)
(cid:12)
(cid:12)
(cid:12)
(cid:12)RDgen(πSgen) − (cid:98)RSgen(πSgen)
(cid:12)
(cid:12)
(cid:12) .

(16)

≤ CDTV(D, DM ) + CDTV(DM , Dgen) +

= C (DTV(D, DM ) + DTV(DM , Dgen)) +

This finishes the proof.

E PROOF OF LEMMA 4.5

Proof. Considering the Markov chain M (p) → Sgen → W , according to the properties of mutual
information, we have:

Furtherly, the following inequality can be derived:

H(Sgen) ≤ H(M (p)).

I(Sgen, W ) ≤ I(M (p), W ).

Building upon equation (18), we can derive the following equations:

I(Sgen, W ) = I(M (p), W ) − δϵ

≤ I(M (p), W ),

(17)

(18)

(19)

where δϵ is the information loss due to the noise ϵ in the data curation process.

Since h(·) and g(·) are deterministic functions which decrease the entropy of random variables, we
have:

Accordingly, the following inequalities can be derived:

H(h(ep)) ≤ H(ep), H(g(eM )) ≤ H(eM ).

I(h(ep), W ) = H(h(ep)) − H(h(ep)|W )

≤ H(ep) − H(ep|W )
= I(ep, W ).

17

(20)

(21)

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Under review as a conference paper at ICLR 2025

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Similarly, we have:

I(g(eM ), W ) = H(g(eM )) − H(g(eM )|W )

≤ H(eM ) − H(eM |W )
= I(eM , W ).
This is because the deterministic functions h(·) and g(·) decrease the information content, and make
the information a subset of the original random variables.

(22)

Then we consider the upper bound of I(M (p), W ) according to the result above:

I(M (p), W ) = I(h(ep) + g(eM ), W )

≤ I(h(ep), W ) + I(g(eM ), W )
≤ I(ep, W ) + I(eM , W )

(23)

For further analysis, we consider the following assumption related to the efficiency of the model
utilizing the prompt:

Lemma E.1. (Efficiency of the model prompting.) For the model M utilizing the prompt p, with
λ ≥ 1, we have:

H(ep) ≤ λI(ep, M (p)).

(24)

Lemma E.1 indicates that the entropy of ep is upper bounded by the mutual information between
the synthetic factor ep and the model output M (p) by a factor of λ. In other words, the efficiency
of the model utilizing the prompt is reflected in the value of λ, which quantifies the extent to which
the model can leverage the information contained in the prompt. For example, a larger λ indicates a
smaller I(ep, M (p)), which implies that the M (p) contains more information from the prompt p.

Building upon Lemma E.1, we can further derive the deduction following equation (23):

I(M (p), W ) ≤ I(ep, W ) + I(eM , W )

= H(ep) − H(ep|W ) + I(eM , W )
= H(M (p)) − H(M (p)) + H(ep) − H(ep|W ) + I(eM , W )
≤ −H(M (p)) + I(ep, M (p)) − I(ep, M (p))

+ λI(ep, M (p)) + H(M (p)) − H(ep|W ) + I(eM , W )

≤ −∆I + I(eM , W ) + H(M (p)) − H(ep|W ) + (λ − 1)I(ep, M (p))
≤ −∆I + Bsyn + H(eM ).

(25)

Lemma E.2. (Entropy gap upper bound) The difference between the entropy of M (p) and ep can
be upper bounded by the following inequality:

H(M (p)) − H(ep) ≤ H(eM ).

The proof of Lemma E.2 is listed in equation (27):

H(M (p)) − H(ep) = H(h(ep) + g(eM )) − H(ep)

≤ H(e(p)) + H(g(eM )) − H(ep)
≤ H(ep) + H(eM ) − H(ep)
= H(eM ).

(26)

(27)

Building upon Lemma E.2, we can further deduce the following inequality following equation (25):

I(M (p), W ) ≤ −∆I + I(eM , W ) + H(M (p)) − H(ep|W ) + (λ − 1)I(ep, M (p))

≤ −∆I + Bsyn − I(ep, W ) + H(M (p)) − H(ep|W ) + (λ − 1)I(ep, M (p))
= −∆I + Bsyn + H(M (p)) − H(ep) + (λ − 1)I(ep, M (p))
≤ −∆I + Bsyn + H(eM ) + (λ − 1)I(ep, M (p)).

(28)

Together with equations (19) and (28), we have:
I(Sgen, W ) = I(M (p), W ) − δϵ

≤ −∆I + Bsyn + H(eM ) + (λ − 1)I(ep, M (p)) − δϵ
≤ −∆I + Bsyn + H(eM ) + δϵ,p,

(29)

18

Under review as a conference paper at ICLR 2025

where δϵ,p = (λ − 1)I(ep, M (p)) − δϵ.

This finishes the proof.

F PROOF OF THEOREM 4.10

Proof. Considering the Markov chain h(ep) → M (p) → Sgen, we have:

H(M (p)) ≥ H(Sgen).

In addition, according to the properties of mutual information, we have:

I(Sanchor, M (p)) ≥ I (h(ep), M (p)) .

Building upon the inequalities above, we can derive the following equations:

∆I = H(M (p)) − I (h(ep), M (p))
≥ H(Sgen) − I(Sanchor, M (p))
= I(Sgen, W ) + H(Sgen|W ) − I(Sanchor, M (p)).

Based on the assumptions mentioned above, we also have:

I(Sanchor; W

′

) = H(Sanchor) − H(Sanchor|W

)
= H(Sanchor) − αH(Sanchor|W )
= I(Sanchor, W ) + (1 − α)H(Sanchor|W ).

′

Furthermore, based on the definitions, we have:

I(Sanchor, M (p)) = H(Sanchor) − H(Sanchor|M (p))

= I(Sanchor, W ) + H(Sanchor|W ) − H(Sanchor|M (p))
= I(Sanchor, W ) + ϵW,p.

By the definition of GGMI, and with equation (33), the following result can be deduced:

GGMI =I(Sanchor, W

′

) − I(Sgen, W )

=I(Sanchor, W ) + (1 − α)H(Sanchor|W ) − I(Sgen, W )
=I(Sgen, W ) + H(Sgen|W ) − I(Sanchor, M (p))

− I(Sgen, W ) − H(Sgen|W ) + I(Sanchor, M (p))
+ I(Sanchor, W ) + (1 − α)H(Sanchor|W ) − I(Sgen, W ).

Subsequently, together with equations (32) and (34), we can further deduce that:

GGMI ≤∆I − 2I(Sgen, W ) − H(Sgen|W ) + I(Sanchor, W )

+ (1 − α)H(Sanchor|W ) + I(Sanchor, M (p))

=∆I − 2I(Sgen, W ) − H(Sgen|W )

+ 2I(Sanchor, W ) + (1 − α)H(Sanchor|W ) + ϵW,p

=∆I − 2H(Sgen) + H(Sgen|W )

+ 2H(Sanchor) − (α + 1)H(Sanchor|W ) + ϵW,p.

(30)

(31)

(32)

(33)

(34)

(35)

(36)

Finally, together with all the deduce and definition above, we have:

GGMI ≤ ∆I − (α + 1)H(Sanchor|W ) + 2∆H + H(Sgen|W ) + ϵW,p,

(37)

This finishes the proof.

19

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Under review as a conference paper at ICLR 2025

G EXPERIMENTS: EXPLORING BETTER SYNTHETIC DATA IN PRACTICE

To further investigate the process of synthetic data generation in real-world settings, we conduct
experiments to evaluate the quality of synthetic data produced under different conditions and aim to
identify the factors that contribute to its effectiveness in enhancing model performance.

The experiments follow the same setup as described in the main text, with the synthetic data Sgen
generated from a generative model M prompted by p. We utilize a standard in-context learning
(ICL) framework to determine p using anchor data Sanchor, and we then evaluate the performance
of the model trained on the synthetic data. Additionally, we estimate key components from our
theoretical analysis in the main text, including information gain ∆I and the entropy of the synthetic
data H(Sgen).

In the remainder of this section, we commence by introducing the experimental setup and the eval-
uation metrics. We then present the results of the synthetic data generation, focusing on the per-
formance of the model trained on synthetic data to assess its quality. Furthermore, we estimate the
key components from our theoretical analysis and analyze the factors that contribute to the effec-
tiveness of synthetic data in improving model performance. Finally, we provide a brief conclusion
and discuss potential principles for generating higher-quality synthetic data in practice.

G.1 EXPERIMENTAL SETUP

We conducted experiments to evaluate the effectiveness of synthetic data generated by a generative
model M prompted by p in enhancing model performance. Our experimental setup follows the
synthetic data utilization process described in the main text, including selecting benchmark dataset,
determining prompt p, generating synthetic data Sgen, training the model on the synthetic data, and
evaluating the trained model.

G.1.1 BENCHMARK DATASET

The benchmark dataset is utilized to sample Sanchor. Specifically, we adopt Dolly-15K (Conover
et al., 2023) as our benchmark dataset, which contains 15,000 lines of text data designed for
instruction-following tasks. We split the benchmark dataset into training and testing sets with a
ratio of 8:2, and Sanchor is sampled from the training set. For each data instance, we retain the keys
“instruction”, “context” and “response” and combine them using the following template.

G.1.2 DETERMINING PROMPT p

Consistent with the methodology described in the main text, we employ a standard In-Context Learn-
ing (ICL) framework to determine the prompt p. Specifically, p = E(Sanchor), where E is a prede-
fined template for the prompt. We follow the settings of Alpaca (Taori et al., 2023) and modify the
template to better suit the benchmark dataset used in our experiments. The modified template is as
follows:

20

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Under review as a conference paper at ICLR 2025

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

You are asked to come up with a set of 20 diverse task instructions. These task instructions will be
given to a language model and we will evaluate the language model for completing the instructions.
Here are the requirements: 1. Try not to repeat the verb for each instruction to maximize diversity.
2. The language used for the instruction also should be diverse. For example, you should combine
questions with imperative instrucitons. 3. The type of instructions should be diverse. The list should
include diverse types of tasks like open-ended generation, classification, editing, etc. 4. The language
model should be able to complete the instruction. For example, do not ask the assistant to create any
visual or audio output. For another example, do not ask the assistant to wake you up at 5pm or set
a reminder because it cannot perform any action. 5. The instructions should be in English. 6. The
instructions should be 1 to 2 sentences long. Either an imperative sentence or a question is permitted.
7. You should generate an appropriate input to the instruction. The input field should contain a specific
example provided for the instruction. It should involve realistic data and should not contain simple
placeholders. The input should provide substantial content to make the instruction challenging but
should ideally not exceed 100 words. 8. Not all instructions require input. For example, when a
instruction asks about some general information, ”what is the highest peak in the world”, it is not
necssary to provide a specific context. In this case, we simply put “noinput” in the input field. 9. The
output should be an appropriate response to the instruction and the input. Make sure the output is less
than 100 words.
Your output should consist of 3 parts: instruction, context and reference response. “Instruction” is the
task instruction which the language model should complete. “Context” is information related to the
instruction, if don’t need, you can set it as empty. “Reference response” is the correct answer to the
instruction your recommend.
Your output must be in the following json form like: {“instruction”: [the instruction you generate],
“context”: [the context you generate], “reference response”: [the reference response you generate]}
Here are some examples you should emulate:
{anchor data}
List of 20 tasks:”’

We then sample Sanchor from the benchmark dataset and populate the “{anchor data}” placeholders
in the prompt template with these samples. This completes the process of determining the prompt p.

G.1.3 GENERATING SYNTHETIC DATA SGEN

After determining the prompt p, we generate synthetic data Sgen by prompting the generative model
M with p. In our experiments, we primarily utilize GPT-4o (OpenAI, 2024) as the generative model
M . Additionally, wo also employ the latest Llama 3.2 models (Meta, 2024) including Llama-3.2-
1B-Instruct and Llama-3.2-3B-Instruct for comparison experiments.

G.1.4 TRAINING ON SYNTHETIC DATA

We fine-tune a GPT-2 (Radford et al., 2019) model using both the synthetic data Sgen generated by
the generative model M and the training set T of the benchmark dataset. The training procedure
follows the standard instruction tuning process, where we fine-tune the model on the synthetic data
for a fixed 20 epochs.

G.1.5 EVALUATING FINE-TUNED MODEL

We assess the performance of the fine-tuned model on the testing set of the benchmark dataset.
Following the evaluation procedure of Zheng et al. (2023c), we evaluate the model’s ability by
rating the generated responses using a LLM. To better align our evaluation with our datasets, we
modify the original evaluation prompt to ensure that the judge LLM compares the output with the
ground-truth answer. The evaluation prompt we adopt is as follows:

21

Under review as a conference paper at ICLR 2025

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Base

Rating

0.1409

Synthetic Data Fine-Tuned
10-ins
3-ins
0.1965
0.1863

20-ins
0.2015

Real Data
Fine-Tuned
0.2745

Table 1: Average ratings of the fine-tuned model on the testing set. The ratings were normalized
using a softmax function. The synthetic data were generated by GPT-4o with varying numbers of
instances in In-Context Learning (ICL) (denoted as x-ins). The unfine-tuned base model (Base) and
the model fine-tuned on real data are marked gray.

Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant to
the user question displayed below. You are provided with 4 parts of the text, [Question] is the question
asked by the user, [Context] is information related to the question, [Reference Answer] is the correct
answer for your reference, and Assistant’s Answer which is surrounded by [The Start of Assistant’s
Answer] and [The End of Assistant’s Answer] is the answer given by the assistant. Your evaluation
should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of
detail of the response. Begin your evaluation by providing a short explanation. Be as objective as
possible. After providing your explanation, you must rate the Assistant’s Answer on a scale of 1 to 10
by strictly following this format: “[[rating]]”, for example: “Rating: [[5]]”.
[Question] {instruction}
[Context] {context}
[Reference Answer] {reference response}
[The Start of Assistant’s Answer] {generated response} [The End of Assistant’s Answer]

We then populate the placeholders “{instruction}”, “{context}”, “{reference response}”, and
“{generated response}” in the evaluation prompt with the corresponding text. We adopt Llama-
3.1-Nemotron-70B-Instruct-HF (Wang et al., 2024) as the judge LLM and extract the ratings from
its output. The final rating is averaged over the testing set to evaluate the performance of the fine-
tuned model.

G.2 SYNTHETIC DATA QUALITY

We assess the quality of synthetic data generated by the generative model M prompted by p in
terms of its effectiveness in enhancing model performance. Specifically, we utilize GPT-4o as M to
generate synthetic data with varying numbers of instances in ICL, corresponding to different sizes
of Sanchor, denoted as 3-ins, 10-ins, and 20-ins, respectively. We then fine-tune a GPT-2 model on
both the synthetic data and the training set of the benchmark dataset. The performance of the fine-
tuned model on the testing set is used as a measure of the quality of the synthetic data. For better
presentation, we apply a softmax function to normalize the ratings. The results are shown in Table 1.

The results demonstrate that the synthetic data effectively enhances the performance of the fine-
tuned model, with the rating positively correlated with the number of instances in ICL. This finding
indicates that appropriately increasing the number of instances in ICL can improve the quality of
the synthetic data. This phenomenon may be attributed to the fact that increasing the number of
instances in the ICL prompts provides the generative model with a richer and more diverse context.
This enhanced context allows the model to capture a broader range of patterns present in the anchor
data, thereby generating synthetic data with richer content.

G.3 ESTIMATING THEORETICAL COMPONENTS

Building upon the results of synthetic data quality, we further estimate the key components from our
theoretical analysis, including information gain ∆I and the entropy of the synthetic data H(Sgen).
We aim to analyze the factors that contribute to improving the quality of synthetic data.

G.3.1 ESTIMATING INFORMATION GAIN

Given the definition of information gain ∆I in Definition 4.2, it is difficult to directly estimate ∆I in
practice. However, it is possible to estimate I(T, Sgen), the mutual information between the synthetic

22

Under review as a conference paper at ICLR 2025

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Sgen
HSIC w/ T (×10−3)

3-ins
7.8703

10-ins
7.8668

20-ins
7.8502

Table 2: The HSIC value between the synthetic data Sgen and training set T for different numbers of
instances in ICL setting.

Sgen
Semantic Entropy

3-ins
1.0739

10-ins
1.0503

20-ins
1.0005

Table 3: The semantic entropy of the synthetic data Sgen in different numbers for instances in ICL
setting.

data Sgen and the training set T of the benchmark dataset where Sanchor is sampled. Since the crucial
part of prompt p is Sanchor, I(T, Sgen) has a negative correlation with ∆I to a certain extent.

To measure I(T, Sgen), we follow the setting of existing works (Qian et al., 2024; Ma et al., 2020)
and utilize HSIC (Gretton et al., 2005) as an estimator. The result is shown in Table 2.

It is supurising that more instances doesn’t increase the HSIC value, but even lead to a lower HSIC
value, indicating reduced mutual information between the synthetic data and the training set. This
phenomenon suggests that enlarging the sizes of Sanchor does not significantly increase the depen-
dency between the synthetic data and the training set, and may even enhance the diversity of the
synthetic data. This may be attributed to the fact that when a LLM with a wide range of knowledge
is employed as M , it leverages its broad understanding to generate synthetic data that is less reliant
on the specific instances in Sanchor. As the number of instances in the ICL setup increases, the LLM
interprets this as a richer and more varied context, thereby increasing the output diversity instead.

A smaller HSIC value indicates a lower mutual information between the synthetic data and the
training set, which leads to a larger information gain ∆I. With Theorem 4.7 and Theorem 4.10, this
guarantees a tighter upper bound of the generalization error and higher GGMI, which contributes to
the quality of synthetic data and increase the generalization capabilities.

G.3.2 ESTIMATING ENTROPY OF SYNTHETIC DATA

As another important component in Theorem 4.10, H(Sgen) is crutial for determining the value of
∆H. We use semantic entropy (Farquhar et al., 2024) as an estimator to measure the entropy of the
dataset and estimate the value of H(Sgen). The result is shown in Table 3.

The results indicate that the semantic entropy of the synthetic data Sgen is also negatively correlated
with the number of instances in ICL. This suggests that increasing the sizes of Sanchor when utilizing
LLM as generative model M can help reduce the entropy of the synthetic data. This reduction
in entropy may be attributed to the richer and more varied context provided by a larger Sanchor,
which enables M to generate more accurate and informative synthetic data, thereby increasing the
faithfulness of the synthetic data.

A smaller semantic entropy indicates a lower entropy of the synthetic data Sgen, which leads to a
larger ∆H. With Theorem 4.10, this benifts increasing the upper bound of GGMI, and contributes
to the generalization capabilities of the model trained on the synthetic data.

G.4 ESTIMATING ON DIFFERENT MODEL ARCHITECTURES

To further investigate the impact of different model architectures and parameters on the quality of
synthetic data, we conduct experiments to evaluate the HSIC value and semantic entropy of the
synthetic data Sgen generated by different models. Due to computational resource limitations, we
utilized GPT-4o, Llama-3.2-3B-Instruct, and Llama-3.2-1B-Instruct as the generative model M to
generate synthetic data with 3 instances in ICL setting. The results are presented in Table 4.

Note that under the prompt determined in the experimental setups, the Llama-3.2-1B-Instruct model
did not adhere to the format requirements and failed to produce meaningful synthetic data. Conse-
quently, the estimators are not available for this model. This observation underscores a fundamental

23

Under review as a conference paper at ICLR 2025

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

HSIC w/ T (×10−3)
Semantic Entropy

GPT-4o Llama-3.2-3B-Instruct Llama-3.2-1B-Instruct
7.8703
1.0739

11.4306
2.9427

/
/

Table 4: The HSIC value and semantic entropy of the synthetic data Sgen generated using different
model architectures. All the synthetic data are generated with 3 instances in ICL setting. Note
that the Llama-3.2-1B-Instruct model did not adhere to the format requirements and thus failed to
produce meaningful synthetic data.

premise that the generative model M must possess sufficient instruction-following capabilities to
generate synthetic data that can be effectively utilized to enhance model performance.

On the other hand, although Llama-3.2-3B-Instruct produced usable synthetic data, its quality was
insufficient for fine-tuning GPT-2, and the HSIC value and semantic entropy were significantly
higher than those of GPT-4o. This may be attributed to the smaller model size of Llama-3.2-3B-
Instruct compared to GPT-4o, resulting in a diminished capacity to generate synthetic data that is
both faithful to and diverse from the anchor data. For instance, we provide some examples of the
synthetic data generated by Llama-3.2-3B-Instruct in the following as a case study:

Instructions generated by Llama-3.2-3B-Instruct:
“instruction”: “Explain the concept of blockchain in simple terms.”
“instruction”: “Explain the concept of artificial intelligence in simple terms.”
“instruction”: “Explain the concept of climate change in simple terms.”
· · ·
“instruction”: “Identify the type of music genre: classical or jazz: ’Moonlight Sonata’ or ’Take Five’”
“instruction”: “Identify the type of literary device used in the following sentence: ’The eyes that fixed
upon her were like two bright stars in the night sky.’”
“instruction”: “Identify the type of music instrument: string or woodwind: ’Violin’ or ’Flute’”
· · ·
“instruction”: “Write a short story about a character who discovers a hidden world within their reflec-
tion.”
“instruction”: “Write a review of the movie ’The Shawshank Redemption’.”
“instruction”: “Write a poem about the beauty of nature.”

The examples demonstrate that the synthetic data generated by Llama-3.2-3B-Instruct is highly ho-
mogeneous, even within a single generation cycle. Moreover, it is highly dependent on the specific
instances in the anchor data, leading to a higher HSIC value. Furthermore, although the synthetic
data lacks diversity in form, the semantic entropy remains high. This indicates that the generated
synthetic data lacks sufficient faithfulness. Collectively, these factors contribute to the poor quality
of the synthetic data produced by Llama-3.2-3B-Instruct.

G.5 CONCLUSION

Building upon the experiments, we can derive some brief conclusions about how to guarantee the
synthetic data quality and estimate the key factors in real-world LLM practice.

The quality of synthetic data is mainly reflected in two aspects: diversity and faithfulness. Diversity
makes the synthetic data contain richer contents and thus increase the information gain ∆I. With our
theoretical analysis, this will benifit the generalization ability of the model post-trained on synthetic
data. Faithfulness makes the synthetic data semantically continuous, and thus decrease the entropy
of the synthetic data Sgen, which also strenghten the generalization capabilities.

In practice, the diversity and the faithfulness can be estimated by HSIC value and the semantic
entropy, respectively, as demonstrated in the experimental settings of this section. It is also important
to highlight that employing a generative model with stronger instruction-following capabilities and
more diverse knowledge can enhance the quality of synthetic data in both aspects.

24

