Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data
Under review as a conference paper at ICLR 2025

SYNTHIO: AUGMENTING SMALL-SCALE AUDIO CLAS-
SIFICATION DATASETS WITH SYNTHETIC DATA

Anonymous authors
Paper under double-blind review

ABSTRACT

We present Synthio, a novel approach for augmenting small-scale audio1 classi-
fication datasets with synthetic data. Our goal is to improve audio classification
accuracy with limited labeled data. Traditional data augmentation techniques,
which apply artificial transformations (e.g., adding random noise or masking seg-
ments), struggle to create data that captures the true diversity present in real-world
audios. To address this shortcoming, we propose to augment the dataset with
synthetic audio generated from text-to-audio (T2A) diffusion models. However,
synthesizing effective augmentations is challenging because not only should the
generated data be acoustically consistent with the underlying small-scale dataset,
but they should also have sufficient compositional diversity. To overcome the first
challenge, we align the generations of the T2A model with the small-scale dataset
using preference optimization. This ensures that the acoustic characteristics of
the generated data remain consistent with the small-scale dataset. To address the
second challenge, we propose a novel caption generation technique that leverages
the reasoning capabilities of Large Language Models to (1) generate diverse and
meaningful audio captions and (2) iteratively refine their quality. The generated
captions are then used to prompt the aligned T2A model. We extensively evaluate
Synthio on ten datasets and four simulated limited-data settings. Results indicate
our method consistently outperforms all baselines by 0.1%-39% using a T2A model
trained only on weakly-captioned AudioSet.

1

INTRODUCTION

Audio classification is the foundational audio processing task of understanding the input audio and
assigning it to one or multiple predefined labels. However, training audio classification models
requires a lot of high-quality labeled data, which is not always readily available (Ghosh et al., 2022).
Manually collecting and annotating large-scale audio datasets is an expensive, time-consuming, and
noisy process (Nguyen et al., 2017; Mart´ın-Morat´o & Mesaros, 2021), and recent concerns about
data privacy and usage rights further hinder this process (Ren et al., 2023).

Data augmentation, which involves expanding original small-scale datasets with additional data,
is a promising solution to address data scarcity. Traditional augmentation techniques attempt to
diversify audio samples by applying randomly parameterized artificial transformations to existing
audio. These methods include spectral masking (Park et al., 2019), temporal jittering (Nanni et al.,
2020), cropping (Niizumi et al., 2021), mixing (Seth et al., 2023; Ghosh et al., 2023b; Niizumi et al.,
2021) and other techniques (Saeed et al., 2021; Al-Tahan & Mohsenzadeh, 2021; Manocha et al.,
2021). While these approaches have shown success, they operate at the level of observed data rather
than reflecting the underlying data-generating process that occurs in real-world scenarios. As a result,
they statistically modify the data without directly influencing the causal mechanisms that produced it,
leading to high correlations between augmented samples and limited control over diversity.

Generating synthetic data from pre-trained text-to-audio (T2A) models addresses the limitations
of standard data augmentation techniques while retaining their strengths of universality, controlla-
bility, and performance (Trabucco et al., 2024). The recent success of generative models makes
this approach particularly appealing (Long et al., 2024; Evans et al., 2024b). However, generat-
ing synthetic audio presents unique challenges due to the complexity of waveforms and temporal

1We use “audio” to refer to acoustic events comprising non-verbal speech, non-speech sounds, and music.

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

dependencies (Ghosh et al., 2024b). We highlight the 3 main challenges in generating effective
synthetic data for audio classification: i) Consistency with the original data: Synthetic audio that
does not align acoustically with the original dataset can hinder effective augmentation and may cause
catastrophic forgetting (Geiping et al., 2022). This misalignment includes spectral, harmonic, and
other inherent acoustic characteristics not easily controlled through prompts. Maintaining consistency
with T2A models trained on internet-scale data remains a challenge, and standard fine-tuning can
often lead to overfitting (Weili et al., 2024). ii) Diversity of generated data: Ensuring compositional
diversity in the generated synthetic data (e.g., sound events, temporal relationships, background
elements, etc.) is critical for effective augmentation. Additionally, a lack of diversity can lead to poor
generalization and learning of spurious correlations, impacting performance. Simple, hand-crafted
prompts (e.g., “Sound of a metro”) often result in repetitive patterns, and creating diverse, meaningful
prompts is labor-intensive. Complex prompts can generate audios that do not preserve the original
label. iii) Limitations of current T2A models: T2A models often struggle to generate diverse audios
and follow details in prompts. This is largely due to the lack of large-scale, open-source datasets for
training, as well as the inherent complexity of non-speech audio domains (Ghosal et al., 2023). These
limitations highlight the need for more advanced approaches for synthetic data generation in audio.

Our Contributions. To address these challenges, we propose Synthio, a novel, controllable and
scalable approach for augmenting small-scale audio classification datasets with synthetic data.
Our proposed approach has 2 main steps:
i)
Aligning the Text-to-Audio Models with Prefer-
ence Optimization: To generate synthetic audios
with acoustic characteristics consistent with the
small-scale dataset, we introduce the concept
of aligning teaching with learning preferences.
Specifically, we align the generations of the T2A
model (acting as the teacher) with the target char-
acteristics of the small-scale dataset using pref-
erence optimization. This approach ensures that
the synthetic audios reflect the acoustic prop-
erties of (or sound similar to) the downstream
dataset, enabling the classification model (the
student) to perform well on test data with sim-
ilar characteristics. To achieve this, we train a
diffusion-based T2A model with preference op-
timization, where audios generated from Gaus-
sian noise are treated as losers and audios from
the downstream dataset are treated as winners.
ii) Generating Diverse Synthetic Augmentations: To generate diverse audios for augmentation, we
introduce the concept of language-guided audio imagination and imagine novel acoustic scenes with
language guidance. Specifically, we generate diverse audio captions that are then used to prompt T2A
models to generate audios with varied compositions. To achieve this, we propose MixCap, where
we prompt LLMs iteratively to generate captions combining existing and new acoustic components.
Additionally, we employ a self-reflection module that filters generated captions and prompts the LLM
to revise those that do not align with the intended label. To summarize, our main contributions are:
1. We introduce Synthio, a novel data augmentation approach for audio classification that
expands small-scale datasets with synthetic data. Synthio uses novel methods to tackle the
inherent challenges of producing consistent and diverse synthetic data from T2A models.
2. We evaluate Synthio across 10 datasets in 4 simulated low-resource settings, demonstrating
that, even with a T2A model trained on weakly captioned AudioSet, Synthio outperforms
all baselines by 0.1%-39%.

Figure 1: Performance comparison of Synthio with
other augmentation methods on down-sampled ESC-
50 (100 samples). Traditional augmentation, such as
SpecAug, degrades performance on small-scale datasets.
Naive synthetic augmentation outperforms traditional
methods significantly but plateaus with higher sample
counts. Synthio further enhances performance by gener-
ating consistent and diverse synthetic data.

3. We conduct an in-depth analysis of the generated augmentations, highlighting Synthio’s
ability to produce diverse and consistent data, its scalability, and its strong performance on
complex tasks such as audio captioning.

2 RELATED WORK

Data Augmentation for Audio and Beyond. Expanding or augmenting small-scale datasets with
additional data has been widely studied in the literature. Traditional augmentation methods, which

2

Classification Accuracy0.40.50.60.70.80.90100400500No AugmentationSpecAugVanilla Syn. Aug.Synthio (ours)  200 300 Number of Generated AugmentationsUnder review as a conference paper at ICLR 2025

apply randomly parameterized artificial transformations to data during training, remain the most
common approach across language Wei & Zou (2019); Karimi et al. (2021), vision (Shorten &
Khoshgoftaar, 2019; Wang et al., 2017; Yun et al., 2019), and audio (Park et al., 2019; Spijkervet,
2021). For audio, specific techniques include SpecAugment, adding background noise, reverberation,
and random spectrogram transformations. With the emergence of generative models, synthetic
data augmentation has been increasingly adopted for language (Ghosh et al., 2023a; 2024c; Chen
et al., 2021) and vision (Trabucco et al., 2024; Zhao et al., 2024), proving to be more effective than
traditional methods. These approaches generally incorporate explicit steps to ensure the consistency
and diversity of generated augmentations. In contrast, application of synthetic data to audio and
speech remain underexplored. Recent attempts include generating synthetic captions for improving
audio-language pre-training (Xu et al., 2023), improving T2A models with synthetic captions (Kong
et al., 2024) and environmental scene classification (Ronchini et al., 2024; Feng et al., 2024).

Few- and Zero-Shot Audio Classification. Few-shot audio classification focuses on training models
to classify audio samples with very limited labeled data per class, often leveraging transfer learning
or meta-learning approaches (Zhang et al., 2019; Wang et al., 2021; Heggan et al., 2022). In contrast,
zero-shot audio classification enables models to generalize to unseen categories without direct training
on those classes, relying on learned representations or external knowledge (Xie & Virtanen, 2021;
Elizalde et al., 2023). Synthetic data research complements these by generating additional labeled
data, improving model performance under low-resource settings while addressing data scarcity
without directly requiring labeled instances from the target categories.

Text-to-Audio Generation. In recent years, there has been a significant surge in research on text-
to-audio (T2A) models. The most popular architectures include auto-regressive models based on
codecs (Kreuk et al., 2023; Copet et al., 2024) and diffusion models Liu et al. (2023); Ghosal
et al. (2023); Evans et al. (2024a). Clotho (Drossos et al., 2020) and AudioCaps (Kim et al.,
2019) remain the largest human-annotated datasets for training these models. However, large-scale
datasets for T2A model training are still scarce. Recently, Yuan et al. (2024) synthetically captioned
AudioSet (Gemmeke et al., 2017), demonstrating its effectiveness for training T2A models. For
downstream adaptation, earlier works have primarily relied on Empirical Risk Minimization (ERM).
Majumder et al. (2024) introduced preference optimization for T2A models, creating a synthetic
preference dataset based on scores provided by a CLAP model (Elizalde et al., 2023).

3 BACKGROUND

Diffusion Models. Diffusion models consist of two main processes: a forward process and a reverse
process. Given a data point x0 with probability distribution p(x0), the forward diffusion process
gradually adds Gaussian noise to x0 according to a pre-set variance schedule γ1, · · · , γT and degrades
the structure of the data. We request readers to refer to App. A.1 for more details on diffusion models.

Reward Modeling. Estimating human preferences for a particular generation x0 (hereafter treated as
a random variable for language), given the context c, is challenging because we do not have direct
access to a reward model r(c, x0). In our scenario, we assume only ranked pairs of samples are
available, where one sample is considered a “winner” (xw
0) under the
same conditioning c. Based on the Bradley-Terry (BT) model, human preferences can be modeled as:
0|c) = σ(r(c, xw
(1)
where σ represents the sigmoid function. The reward model r(c, x0) is parameterized by a neural
network ϕ and trained through maximum likelihood estimation for binary classification:

0 ) and the other a “loser” (xl

0 ) − r(c, xl

pBT(xw

0 ≻ xl

0))

Here, prompt c and data pairs (xw

LBT(ϕ) = −E
0 , xl

0 ,xl
0

(cid:2)log σ(rϕ(c, xw

0 ) − rϕ(c, xl
c,xw
0) are drawn from a dataset labeled with human preferences.

0))(cid:3)

(2)

RLHF : (Christiano et al., 2017) The goal of RLHF is to optimize a conditional distribution pθ(x0|c),
where c ∼ Dc, such that the latent reward model r(c, x0) is maximized. This is done while regulariz-
ing the distribution through the Kullback-Leibler (KL) divergence from a reference distribution pref,
resulting in the following objective:

max
pθ

Ec∼Dc,x0∼pθ(x0|c)[r(c, x0)] − βDKL[pθ(x0|c)∥pref(x0|c)]

(3)

Here, the hyperparameter β controls the strength of regularization.

3

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Under review as a conference paper at ICLR 2025

DPO : DPO directly optimizes the conditional distribution pθ(x0|c) to align data generation with the
preferences observed in (any form of) feedback. The goal is to optimize the distribution of generated
data such that it maximizes alignment with human preference rankings while maintaining consistency
with the underlying reference distribution pref(x0|c).
The optimal solution p∗

θ(x0|c) for the DPO objective can be expressed as:

p∗
θ(x0|c) = pref(x0|c)

exp(r(c, x0)/β)
Z(c)

where Z(c) is the partition function, defined as:

Z(c) =

(cid:88)

x0

pref(x0|c) exp(r(c, x0)/β)

(4)

(5)

This term ensures proper normalization of the distribution, and β controls the regularization, balancing
between adherence to the reference distribution and preference maximization. The reward function
r(c, x0) is then reparameterized as:

r(c, x0) = β log

p∗
θ(x0|c)
pref(x0|c)

+ β log Z(c)

Using this reparameterization, the reward objective can be formulated as:

LDPO(θ) = −E

c,xw

0 ,xl
0

(cid:20)

(cid:18)

log σ

β log

pθ(xw
pref(xw

0 |c)
0 |c)

− β log

(cid:19)(cid:21)

pθ(xl
pref(xl

0|c)
0|c)

(6)

(7)

By optimizing this objective, DPO enables direct preference learning, optimizing the conditional
distribution pθ(x0|c) in such a way that it better reflects human preferences, as opposed to traditional
approaches that optimize the reward function first and then perform reinforcement learning.

DPO for Diffusion Models: Very recently, Wallace et al. Wallace et al. (2024) propose a formulation
for optimizing diffusion models with DPO. The primary issue with optimizing diffusion with DPO is
that the distribution pθ(x0|c) is not tractable due to the need to consider all possible diffusion paths
leading to x0. To address this, Wallace et al. propose to leverage the evidence lower bound (ELBO)
to incorporate latents x1:T , which represent the diffusion path. The reward R(c, x0:T ) accounts for
the entire sequence, leading to the reward function:

r(c, x0) = Epθ(x1:T |x0,c)[R(c, x0:T )]
(8)
Instead of directly minimizing the KL-divergence as typically done, they propose to utlize the
upper bound of the joint KL-divergence DKL[pθ(x0:T |c)||pref(x0:T |c)]. This is integrated into the
optimization objective, enhancing the practicality of training diffusion models with preferences. The
new objective, aiming to maximize the reward and match the distribution of the reverse process of pθ
to the reference model pref, is given by:
max
pθ

Ec,x0∼pθ(x0:T |c)[r(c, x0)] − βDKL[pθ(x0:T |c)||pref(x0:T |c)]

(9)

Training efficiency is improved by approximating the intractable reverse process using a forward
approximation q(x1 : T |x0). The DPO then integrates this into the loss function, which involves
comparing the log likelihood ratio of the probabilities under pθ and pref for winning and losing paths:

LDPO-Diffusion(θ) = −E

(c,xw

0 ,xl

0)∼Dpref

(cid:20)

(cid:18)

log σ

βT log

pθ(xw
pref(xw

1:T |xw
0 )
1:T |xw
0 )

− βT log

(cid:19)(cid:21)

pθ(xl
pref(xl

1:T |xl
0)
1:T |xl
0)

(10)
After applying Jensen’s inequality to take advantage of the convexity of − log σ, we push the
expectation outside, allowing us to simplify the objective. By approximating the denoising process
with the forward process, the final form of the loss for DPO in diffusion models, in terms of the L2
noise estimation losses, becomes:

LDPO-Diffusion(θ) = −E

(11)
0)∼Dpref,t,ϵw
where ∆L is the L2 weighted noise estimation losses between the preferred (winner) and less
preferred (loser) samples.

[log σ (−βT ω(λt)∆L)]

(c,xw

0 ,xl

t ,ϵl
t

4 METHODOLOGY

Let Dsmall = {(ai, li), 1 ≤ i ≤ n} be a high-quality, small-scale human-annotated audio classification
dataset with n audio-label pairs. Let Da-c be a potentially noisy, large-scale weakly-captioned dataset
of audio-caption pairs with zero intersection with Dsmall. Our goal is to train a T2A model T θ using

4

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Under review as a conference paper at ICLR 2025

Da-c, then use it to generate a synthetic dataset Dsyn and then finally add it to Dsmall (now attributed as
Dtrain) to improve audio classification performance. This is accomplished through two key steps: first,
aligning the generations from T θ with the acoustic characteristics of Dsmall, and second, generating
new captions to prompt T θ for creating synthetic audio data.

4.1 ALIGNING THE TEXT-TO-AUDIO MODEL USING PREFERENCE OPTIMIZATION

T2A models trained on internet-scale data often gen-
erate audio that diverges from the characteristics of
small-scale datasets, resulting in distribution shifts.
These mismatches can include variations in spectral
(e.g., frequency content), perceptual (e.g., pitch, loud-
ness), harmonic, or other acoustic characteristics 2.
This misalignment arises from the non-deterministic
nature of T2A generation and it is impractical to pro-
vide detailed attributes (like “loud” or “high-pitched”)
in prompts, as (i) there are no scalable methods for
extracting specific attributes for each label, and (ii)
T2A models struggle with accurately following fine-
grained prompt details (Wang et al., 2024).

Figure 2: We propose to align the T2A model
T θ with the small-scale dataset Dsmall using DPO.
This helps us generate audios with acoustic char-
acteristics aligned to that of Dsmall.

To address these issues, we propose the concept of
aligning teaching with learning preferences. Our ap-
proach assumes that the classification model (viewed
as the student) performs better when trained on syn-
thetic audio that closely matches the inherent acoustic
properties of our high-quality and human-labeled Dsmall. Thus, we align the generations of the T2A
model (viewed as the teacher) to Dsmall, ensuring that the generated augmentations align with the de-
sired characteristics and sound similar, ultimately enhancing the student model’s ability to generalize
to similarly characterized test data. As shown in Fig. 2, we achieve this using preference optimization
(DPO in our case) and align generations of T θ with Dsmall. Unlike standard fine-tuning, which can
lead to less diverse outputs and overfitting due to a narrow focus on minimizing loss, preference
optimization encourages greater exploration in the model’s output space, preventing mode collapse
and fostering more diverse augmentations. Additionally, DPO leverages pairwise learning, offering
richer training signals compared to the independent outputs used in standard fine-tuning, further
mitigating overfitting risks. We detail our two-step approach for DPO optimization below:

1 , al

j , al

1), · · · , (aw

Step 1: Construction of the Preference Dataset. To create our preference dataset Dpref =
{(aw
j)}, we first generate template-based captions for each instance in Dsmall in the
form: “Sound of a label”, where label is the category associated with the audio. For each instance,
we prompt the T2A model j times, with all generations starting from randomly initialized Gaussian
noise (generation configuration is detailed in Section 5). Each generated audio is then paired with
the corresponding ground-truth audio from the gold dataset. This resulting Dpref dataset has n × j
instances, where the generated audio is treated as the “loser” and the ground-truth audio as the
“winner”. This simple approach has proven highly effective in aligning generations by generative
models by prior work (Majumder et al., 2024; Tian et al., 2024).

Step 2: Preference Optimization Using DPO. After constructing Dpref, we train our T2A model
on this dataset with DPO using the approach outlined in Section 3. The resulting aligned model is
referred to as T θ

aln. Details of the hyper-parameters used for training are provided in Section 5.

4.2 GENERATING DIVERSE SYNTHETIC AUGMENTATIONS

It is not well-studied in the literature on how to leverage synthetic audio generation for downstream
tasks. The only existing work relied on manually crafted prompt templates (e.g., “Sound of a
{label}”) (Ronchini et al., 2024). It has a significant limitation: there is no precise control over

2When prompted with “sound of a bus” for the category “bus” in the TUT-Urban dataset, the generated audio
may not reflect the typical bus sounds in European cities (where TUT was recorded), as bus sounds can vary by
region, with some featuring loud engines and dense crowds while others have quieter engines and sparse crowds.

5

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Sound of a {label}Random Gaussian NoiseText ConditionWinning AudiosLosing AudiosText-to-AudioModelAligned Text-to-Audio ModelGeneratedAudiosAdapters Training🔥🔥❄PreferenceOptimizationUnder review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Figure 3: Overview of our proposed Language-Guided Audio Imagination for generating diverse synthetic
augmentations. Starting with the small-scale dataset, we first generate audio captions and use an LLM to extract
acoustic components (Prompt 1). Using these components and audio labels, we prompt the LLM to generate new
and diverse captions (Prompt 2), which are then used to prompt the aligned T2A model for audio generation.
The generated audios are filtered for label consistency using CLAP, with accepted audios added to the final
synthetic dataset. Rejected audios undergo caption revision (Prompt 3) through a self-reflection process, and the
revised captions are used to regenerate audios, iterating this process i times. Example captions are in Table 6.

the specific components in the generated audio for a given caption. This can result in repetitive or
completely inconsistent patterns, particularly with weaker T2A models 3. These could bias the model
to learn spurious correlations, a known issue in synthetic data augmentation (Ghosh et al., 2024c).

While the alignment stage helps the T2A model generate audio with acoustic characteristics similar to
the small-scale dataset (e.g., spectral, harmonic, etc.), it does not fully account for the compositional
diversity of the generated audios (e.g., sound events, their temporal relationships, background
elements). To tackle this, we propose the concept of language-guided audio imagination, where we
propose to imagine novel audios guided by language. Specifically, we leverage the reasoning abilities
of LLMs to generate diverse and meaningful captions for a category label in a controlled yet scalable
manner. These captions are then used to prompt our aligned T2A model for generating novel audios.

4.2.1 GENERATING DIVERSE PROMPTS WITH MIXCAP

We propose MixCap, a prompt generation method that creates diverse and effective captions in
three steps: First, we employ GAMA (Ghosh et al., 2024a) to caption all audio files in Dsmall. Next,
we prompt an LLM to extract phrases describing the acoustic components of the audio. These
components correspond to the acoustic elements such as backgrounds and foreground events, and
their attributes and relations, etc (see prompt in Appendix A.2). Finally, for each training instance
in Dsmall, we prompt the LLM with the ground-truth label and the extracted components from all
instances to generate N diverse audio captions that blend existing and new components.

4.2.2 FILTERING & SELF-REFLECTION

Filtering. After generating captions and their corresponding audio, we filter the audio for label
consistency. While LLMs can generate diverse captions, the audio produced must remain aligned
with the ground-truth label. To ensure this, we use CLAP to evaluate the generated audio, accepting
those that meet a similarity threshold of p% and rejecting the rest. We denote the accepted audios as
Dacc
syn. Our CLAP model is pre-trained on Da-c and we fine-tune the last
layer with Dsmall to adapt to the target dataset. Example captions are in Table 6.
Self-Reflection. For the rejected audios in Drej
syn, we prompt the LLM to reflect on its generated
captions and revise them to better align with the target label. Precisely, we feed the LLM with the

syn and the rejected ones as Drej

3For example, when prompted with “Sound of a park”, we observed that 9 out of 10 times, the model
generated the sound of children playing as part of the generated audio. On the other hand, when prompted with
“Sound of a airport”, the model generates audios with background announcements, which could vary by regions.

6

ASTSmall-ScaleDatasetSyntheticDataLLMGenerated AudiosRejectedAudiosAccepted AudiosSelf-ReﬂectionExisting AcousticComponents     CLAP     FilteringLLMAudio CaptioningModelAudio CaptionsPrompt 1Text-to-AudioModelNew AudioCaptionsPrompt 2Prompt 3MixCap🔥❄❄❄Trainable🔥Frozen❄Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

original caption of each rejected audio along with extracted components from all accepted captions
in Dacc
syn and task it to rewrite the rejected captions. The revised captions are then used to generate new
audio, which is again filtered using CLAP. Audios that meet the threshold are accepted while ones
that don’t go through the process. This repeats for i iterations or until there are no rejected samples.

Fine-tuning for Audio Classification. After the self-reflection stage, the final set of accepted
synthetic audios is denoted as Dsyn, containing ≈ N × n audio-label pairs, where N represents the
augmentation factor (e.g., with 100 gold samples, we generate 100 × N synthetic samples). This set
is then combined with Dsmall to form the final training dataset Dtrain, which is then used to train the
audio classification model.

5 EXPERIMENTAL SETUP

Models and Hyper-Parameters. For our T2A model, we choose the Stable Audio architecture (Evans
et al., 2024b). We train the model from scratch on Sound-VECaps (Yuan et al., 2024) (with ≈1.5
million weakly captioned audio-caption pairs) to avoid any data leakage. For training, we employ a
batch size of 64, an AdamW optimizer, a learning rate of 5e-4, and a weight decay of 1e-3 for 40
epochs. For DPO-based alignment tuning, we generate j = 2 losers and fine-tune with a batch size
of 32 and a learning rate of 5e-4 for 12 epochs. For our audio classification model, we employ the
Audio Spectrogram Transformer (AST) (Gong et al., 2021) (pre-trained on the AudioSet dataset) and
fine-tune it with a batch size of 24 and learning rate of 1e-4 for 50 epochs. For CLAP filtering we
employ p = 0.85. For prompting our diffusion model we use Text CFG=7.0. In each experiment, we
adjust the number of generated augmentations N (ranging from 1 to 5) based on performance on the
validation set. All results are averaged across 3 runs.

Datasets. We create small-scale datasets by downsampling commonly used audio classification
datasets to n samples. Our selected datasets include a mix of music, everyday sounds, and acoustic
scenes. For multi-class classification, we use NSynth Instruments, TUT Urban, ESC50 (Piczak),
USD8K (Salamon et al., 2014), GTZAN (Tzanetakis et al., 2001), Medley-solos-DB (Lostanlen
& Cella, 2017), MUSDB18 (Rafii et al., 2017), DCASE Task 4 (Mesaros et al., 2017), and Vocal
Sounds (VS) (Mesaros et al., 2017), evaluating them for accuracy. For multi-label classification, we
use the FSD50K (Fonseca et al., 2022) dataset and evaluate it using the F macro
metric. We exclude
AudioSet from evaluation as Sound-VECaps is derived from it. To ensure a downsampled dataset that
has a label distribution similar to that of the of the original dataset, we employ stratified sampling
based on categories. Our experiments are conducted with n = {50, 100, 200, 500} samples, and we
downsample the validation sets for training while evaluating all models on the original test splits.

1

Baselines. Our baselines include: (i) Gold-only (No Aug.): We employ only the small-scale
dataset for training and do not perform any augmentations. (ii) Traditional augmentation baselines:
SpecAugment, Noise Augmentation (we either add random Gaussian noise or background noise from
AudioSet and present averaged results), Pitch and Time Shift and Audiomentations (Jordal, 2021)
– a combination of the AddGaussianNoise, TimeStretch, PitchShift, Shift, SpecFrequencyMask,
TimeMask and TimeStretch – combination with the highest average score on 4 datasets and splits
and was selected after grid search over all possible combinations). (iii) Generative baselines: Vanilla
Synthetic Augmentation (Vanilla Syn. Aug.) – we prompt Tθ with template captions), Vanilla Syn.
Aug. + LLM Caps – we prompt Tθ with random captions generated with LLMs. (iv) Finally, inspired
by Burg et al. (2023), we also employ a retrieval baseline where instead of generating augmentations
from our T2A model trained on Da-c, we just retrieve the top-n instances (w.r.t. CLAP similarity)
from the AudioSet for each instance in Dsmall as our augmentations.

Ablations. We ablate Synthio with: (i) w/o Self-Reflection: We remove the repetitive self-reflection
module and iterate and filter only once; (ii) w/o DPO: We skip the tuning step and prompt the
un-alined T θ for augmentations; (iii) w/ ERM: We replace DPO tuning with standard Empirical
Risk Minimization(ERM)-based fine-tuning with diffusion loss; (iv) w/ Template Captions: We
remove MixCap and self-reflection modules and prompt T θ
aln with template captions; (v) w/o
MixCap: Similar to our Random Captions baseline, but we retain all other modules of Synthio.

6 RESULTS AND DISCUSSION

Main Results. Table 1 showcases the performance comparison between Synthio and the baseline
methods. Synthio consistently outperforms all baselines by 0.1%-39%, achieving notable improve-

7

Under review as a conference paper at ICLR 2025

Table 1: Result comparison of Synthio with baselines on 10 datasets and 4 small-scale settings. n refers to the
number of samples in the small-scale dataset augmented with synthetic data. Synthio outperforms our baselines
by 0.1% - 39%. We also highlight the relative improvements by Synthio compared to the Gold-only.

n

Method

ESC-50 USD8K GTZAN Medley

Gold-only (No Aug.)

22.25

55.09

47.05

47.23

TUT

37.60

NSynth

VS

MSDB

DCASE

FSD50K

33.32

77.49

56.85

12.09

13.21
12.93
12.81
13.28
10.53
15.89
13.07

7.16

8.06
10.04
7.93
10.17
7.28
10.63
10.70

17.23+42% 13.91+94%

14.15
13.28
15.63
14.82
14.53

12.50

13.35
12.19
14.17
16.93
10.93
15.73
16.32

13.06
13.79
13.74
12.52
10.13

10.53

13.71
13.11
14.80
13.55
10.05
12.63
13.25

19.38+55% 16.35+55%

16.32
17.21
15.89
16.77
14.83

23.15

13.62
14.52
12.14
13.62
12.53

13.59

Random Noise
Pitch Shifting
SpecAugment
Audiomentations
Retrieval
Vanilla Syn. Aug.
+ LLM Caps.

Synthio (ours)

50

w/ Template Captions
w/ ERM
w/o Self-Reflection
w/o MixCap
w/o DPO

57.42
59.32
58.36
60.13
37.14
63.54
65.84

45.20
46.80
46.00
47.25
42.55
55.35
63.74

18.50
20.55
19.50
20.35
19.20
40.75
36.80

35.86
37.22
36.73
38.24
35.80
41.50
40.90

46.55
48.17
47.18
48.30
43.65
47.23
55.36
49.50+122% 76.12+38% 68.20+44% 60.58+28% 43.84+17% 40.83+22% 80.67+4%
54.52
56.60
58.00
52.18
52.55

32.42
34.34
27.32
28.15
31.27
33.17
38.17

76.41
78.17
77.27
79.12
71.42
78.37
78.77

41.25
41.30
45.25
42.70
36.55

66.11
69.80
72.57
64.72
68.12

37.52
38.62
39.50
36.13
40.31

64.40
61.70
64.55
54.65
56.10

41.37
42.00
42.81
41.93
41.39

78.57
79.75
78.56
78.70
79.03

52.55
54.50
53.25
54.51
51.35
54.10
57.05
60.15+5%
59.60
57.75
57.25
58.80
57.55

Gold-only (No Aug.)

56.75

72.89

64.15

57.81

47.14

39.11

84.32

65.60

Random Noise
Pitch Shifting
SpecAugment
Audiomentations
Retrieval
Vanilla Syn. Aug.
+ LLM Caps.

Synthio (ours)

100

w/ Template Captions
w/ ERM
w/o Self-Reflection
w/o MixCap
w/o DPO

Gold-only (No Aug.)

Random Noise
Pitch Shifting
SpecAugment
Audiomentations
Retrieval
Vanilla Syn. Aug.
+ LLM Caps.

Synthio (ours)

200

w/ Template Captions
w/ ERM
w/o Self-Reflection
w/o MixCap
w/o DPO

Gold-only (No Aug.)

Random Noise
Pitch Shifting
SpecAugment
Audiomentations
Retrieval
Vanilla Syn. Aug.
+ LLM Caps.

Synthio (ours)

500

w/ Template Captions
w/ ERM
w/o Self-Reflection
w/o MixCap
w/o DPO

71.54
73.52
72.43
73.82
68.24
77.31
79.73

58.50
59.55
47.50
48.50
52.45
77.25
67.05

65.50
66.75
69.75
71.05
61.55
68.25
67.90

46.21
47.50
50.07
51.14
45.39
49.96
48.63

56.98
58.46
58.06
59.32
54.83
63.58
65.79
83.35+47% 85.00+17% 71.20+11% 71.23+23% 52.42+11% 44.92+15% 86.70+3%
64.20
66.57
68.52
66.52
60.81

38.20
39.53
41.96
42.15
37.84
42.31
41.83

83.33
85.07
85.14
85.24
83.27
84.78
84.83

78.00
73.20
77.65
73.50
66.75

80.32
81.81
82.38
78.30
75.46

42.76
43.74
44.38
42.27
40.31

68.15
67.25
69.55
68.50
66.15

49.95
51.11
51.75
50.63
48.78

85.11
84.73
82.53
83.52
84.67

66.15
68.25
66.40
68.40
58.55
63.55
65.95
68.80+5%
66.05
68.00
66.20
66.35
67.85

84.75

83.55
84.90
85.10
85.25
82.55
85.40
85.80
86.10+2%
85.95
85.35
84.85
84.95
84.80

90.75

89.55
88.50
89.50
89.95
85.50
91.50
89.90
92.10+2%
91.70
91.20
91.85
91.70
90.15

74.80

75.15
74.48
76.46
75.80
71.20
77.96
78.37

77.00

75.50
78.55
76.25
77.30
73.65
77.10
79.55

67.41

66.71
67.74
65.70
67.00
65.80
78.97
74.14

55.32

54.42
55.44
55.72
55.21
53.25
55.51
54.73

82.81+11% 82.05+7%

79.40+18% 56.83+3%

80.84
79.82
81.97
81.27
76.23

87.88

88.25
88.83
89.01
88.75
84.86
88.18
86.91
89.18+2%
88.93
88.25
88.72
87.93
88.21

79.25
80.20
78.25
79.55
75.30

79.25

78.90
79.75
80.25
81.25
77.25
79.35
79.55
82.25+4%
80.40
79.15
80.15
80.95
79.45

77.56
74.43
75.53
73.50
73.13

75.65

76.01
75.61
76.68
77.66
73.62
77.97
77.91
78.62+4%
76.64
77.38
78.57
76.61
76.03

55.99
55.76
56.39
55.27
55.99

65.72

65.10
64.93
66.74
66.92
62.73
65.93
65.95
67.81+3%
66.47
65.80
66.21
65.91
66.01

48.77

87.38

68.80

47.83
48.12
54.80
53.15
47.63
55.20
56.21

15.32
17.51
17.93
18.36
15.36
19.04
18.14

86.45
87.47
87.42
86.08
86.28
86.49
87.02

24.82
23.11
27.36
26.29
19.51
28.55
28.40

65.45
69.80
69.25
70.50
63.55
72.95
73.16
57.10+17% 87.52+0.2% 80.40+17% 32.81+42% 20.85+53%
74.55
74.40
75.55
78.55
73.15

19.04
18.22
17.28
19.42
17.17

56.33
56.15
56.76
55.54
52.73

87.25
86.92
86.22
85.78
86.52

29.12
29.81
31.13
28.35
26.79

63.47

89.33

72.05

34.30

20.19

64.15
64.59
64.43
65.21
61.44
64.52
64.39
65.40+3%
64.71
64.27
63.89
64.23
63.61

90.15
89.87
90.38
91.34
87.33
90.31
90.09
91.42+2%
90.97
88.74
90.17
90.23
89.83

73.25
72.15
72.95
73.65
70.20
73.25
73.05
74.70+3%
73.35
74.20
72.15
73.40
72.65

37.21
36.54
38.33
38.75
30.17
37.26
38.74
39.24+6%
38.28
38.03
37.97
39.11
37.04

19.49
21.24
21.46
23.11
14.17
23.52
22.67
23.89+18%
22.35
22.39
22.41
21.65
20.19

ments in overall classification accuracy compared to Gold-only. The highest gains are observed on
USD8K, while the least is on Vocal Sound, likely due to the T2A dataset’s heavy representation of
music compared to the more sparse vocal sounds. Performance gains tend to decrease as the number
of gold samples n in Dsmall grows, aligning with observed trends in prior studies. Detailed results on
the full non-down-sampled datasets can be found in Appendix A.4.1. Although Vanilla Synthetic
Augmentations emerge as the strongest baseline, they lag behind Synthio by an average of 3.5%.

Ablations. The most significant performance drop in Synthio is observed w/o DPO, resulting in
an average decline of 4.5%, highlighting the crucial role of consistency in generating effective
augmentations. Second to w/o DPO, the highest drop is seen in w/ Template Captions, with average
decline of 2.7%, thus highlighting the importance of MixCap.

8

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Under review as a conference paper at ICLR 2025

Figure 4: Comparison of spectral and pitch features between generated audios in Dsyn and real audios in Dsmall
(for n = 100). Synthio-generated audios closely replicate the features of real data, demonstrating its ability to
produce augmentations that maintain consistency with the original dataset (also see FAD scores in Sec. A.4.3).

6.1 HOW CONSISTENT AND DIVERSE ARE AUGMENTATIONS GENERATED BY SYNTHIO?

Table 2: CLAP similarity score be-
tween real audios and generated data.
Lower scores show higher composi-
tional diversity among generated augs.

Fig. 4 compares the distributions of pitch and various spectral
features between generated audios in Dsyn and real audios in
Dsmall across different methods on the USD8K and NSynth
datasets. The features analyzed include Pitch Salience (clar-
ity of the main pitch) (Ricard, 2004), Spectral Flatness (tonal
vs. noise-like quality) (Peeters, 2004), Flux (rate of spectral
change) (Tzanetakis & Cook, 1999), and Complexity (level of
sound detail) (Laurier et al., 2010). Notably, Synthio-generated
audios closely replicate the spectral features of the original
audios, showing the best alignment among all methods and
demonstrating Synthio’s ability to generate consistent augmen-
tations. Table 2 presents CLAP similarity scores between ground-truth audios and their N generated
augmentations, averaged across all dataset instances. Audios generated with Synthio achieve the
highest compositional diversity for generated audios among all baselines. Table 8 shows that audios
generated using Synthio have the highest similarity with the ground-truth category label.

w/ Template Captions
w/ ERM

w/ Template Captions
w/ ERM

Vanilla Syn. Aug.
Synthio (ours)

Vanilla Syn. Aug.
Synthio (ours)

USD8K(↓) NSynth(↓)

47.22
34.58
46.84
52.54

45.17
35.09
46.82
50.01

31.76
22.97
33.00
42.33

33.81
23.03
37.16
43.98

Method

100

200

#

6.2 HOW GOOD ARE SYNTHETIC AUDIOS GENERATED BY SYNTHIO?

Consistent with prior findings in vision (He et al.,
2023), we observe that synthetic data alone performs
sub-optimally compared to human-annotated data.
However, our results show that enhancing the consis-
tency and diversity of synthetic data aided by a small-
scale version of the target dataset significantly im-
proves model performance. Table 3 compares models
trained exclusively on synthetic data with our base-
lines (i.e., only Dsyn is used for training AST). Syn-
thio outperforms all baselines by 0.1%-26.25%, with
DPO-based alignment driving the improvements.

Table 3: Performance comparison of Synthio with
baselines on synthetic-only audio classification.

n

Method

GTZAN

VS

TUT MSDB

Gold-only (No Aug.)

Vanilla Syn. Aug.
Synthio (ours)

100

w/ Template Captions
w/ ERM
w/o DPO

Gold-only (No Aug.)

Vanilla Syn. Aug.
Synthio (ours)

200

w/ Template Captions
w/ ERM
w/o DPO

64.15

29.05
33.10
24.50
25.65
17.60

77.00

32.35
35.15
29.90
28.10
19.85

84.32

47.14

34.13
39.20
30.99
32.76
21.57

21.69
24.51
21.73
24.40
20.39

87.38

55.32

41.96
48.14
35.53
36.29
26.85

24.23
27.00
23.61
25.71
21.40

65.60

35.60
56.45
40.40
42.85
30.20

68.80

39.25
61.45
41.20
46.70
36.75

6.3 CAN SYNTHIO BE EXTENDED TO THE MORE COMPLEX AUDIO CAPTIONING TASK?

involves describing the content of an audio sam-
Audio captioning, unlike classification,
ple using natural
To demonstrate Synthio’s
effectiveness for audio captioning, we evaluated it on down-sampled versions of Audio-
Caps.
For this task, we adapted Synthio by removing the audio captioning and CLAP
filtering stages and we extract acoustic features directly from the existing audio captions.

language, making it a more complex task.

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

NSynthUSD8KMethodMethodMethodMethodMethodMethodMethodMethodUnder review as a conference paper at ICLR 2025

Table 4: Performance comparison of Synthio with
baselines on audio captioning.

Additionally, we retrain our T2A model on a modified
version of Sound-VECaps, excluding any audio from
AudioCaps. Training and evaluation were conducted
using the EnCLAP framework (Kim et al., 2024), and
the dataset was expanded with 4× synthetic samples.
As shown in Table 4, Synthio significantly outper-
forms baseline settings, with improvements largely
due to better alignment w/ DPO. However, manual
inspection revealed that generated audios occasionally do not match their captions compositionally,
reflecting limitations of the current T2A model. While this issue does not affect classification, it
poses challenges for captioning. We will explore more advanced methods as part of future work.

Gold-only (No Aug.)
Vanilla Syn. Aug.
VECaps Retrieval
Synthio (ours)

Gold-only (No Aug.)
Vanilla Syn. Aug.
VECaps Retrieval
Synthio (ours)

0.0754
0.0741
0.0550
0.104

METEOR (↑) CIDEr (↑)

0.112
0.140
0.100
0.202

0.127
0.135
0.088
0.185

0.067
0.092
0.068
0.119

0.112
0.136
0.082
0.194

0.125
0.128
0.108
0.169

0.157
0.166
0.097
0.256

0.148
0.157
0.094
0.227

SPIDEr (↑)

SPICE (↑)

Method

1000

500

n

Table 5: Performance comparison of Synthio
with other baselines on different values of N .

6.4 HOW WELL DOES SYNTHIO SCALE?
Table 5 compares the performance of Synthio, SpecAug-
ment, and Vanilla Synthetic Augmentations across differ-
ent scaling factors N = {1, 2, 3, 4, 5}, where N represents
the number of synthetic samples generated per original
sample in the small-scale dataset (in this case we fix n =
100). As observed, SpecAugment, a traditional augmen-
tation method, cannot scale with increasing N , and the
performance of Vanilla plateaus at higher N . A similar
saturation occurs with Synthio when MixCap is not used.
Even without DPO, Synthio maintains better scalability,
though with reduced overall performance. These results
highlight that MixCap’s ability to generate diverse captions is crucial for Synthio’s scalability.

SpecAugment
Vanilla Syn. Aug.
Synthio (ours)
w/o MixCap
w/o DPO

SpecAugment
Vanilla Syn. Aug.
Synthio (ours)
w/o MixCap
w/o DPO

41.96
33.13
35.28
40.41
39.23

47.50
67.90
77.45
64.30
61.55

47.50
77.25
81.75
68.45
64.25

41.96
35.28
36.37
41.08
39.42

41.96
42.31
43.56
41.95
40.17

47.50
76.75
82.55
71.55
65.95

41.96
41.54
44.92
42.27
40.31

47.50
75.60
83.15
72.85
66.60

Dataset Method

Scaling Factor N

NSynth

ESC50

5x

47.50
71.25
83.35
73.50
66.75

41.96
38.27
44.81
42.15
39.82

4x

3x

1x

2x

6.5 DOES SYNTHIO HELP LONG-TAILED CATEGORIES?

Figure 5 shows the classification accuracy
on four underrepresented categories in the
NSynth dataset, comparing performance
before and after applying Synthio aug-
mentations. We selected categories with
the lowest frequency in the downsampled
dataset, such as flute and guitar, which ap-
pear only once in the down-sampled sets.
Synthio significantly boosts accuracy, with
improvements up to 48%. Notably, cat-
egory labels like flute and guitar, which
originally had 0% accuracy, show substan-
tial gains with Synthio augmentation. This
demonstrates Synthio’s effectiveness in boosting performance on long-tail labels, a common challenge
in real-world datasets (Zhang et al., 2023).

Figure 5: Category-wise improvement in performance with
Synthio augmentations for long-tailed categories.

7 CONCLUSION, LIMITATIONS, AND FUTURE WORK

We introduced Synthio, a novel approach for augmenting small-scale audio classification datasets
with synthetic data. Synthio incorporates several innovative components to generate augmentations
that are both consistent with and diverse from the small-scale dataset. Our extensive experiments
demonstrate that even when using a T2A model trained on a weakly-captioned AudioSet, Synthio
significantly outperforms multiple baselines.

However, Synthio has some limitations: (i) Its performance is influenced by the capabilities of the T2A
model and the quality of its training data. As T2A models continue to improve, we expect Synthio’s
performance to benefit accordingly. (ii) The process of generating audio captions using LLMs may
introduce biases inherent in the LLMs into the training process. (iii) Synthio is computationally
more intensive than traditional augmentation methods due to the need for prompting LLMs and
T2A models. We anticipate that ongoing advancements in model efficiency will help mitigate these
computational challenges.

10

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

44.0031.4622.8853.525.5364.3639.0048.0011.7617.9921.1114.62015.34CategoriesClassification Accuracy (%)020406080basskeyboardstringorganfluteguitarreedmalletImproved w/ SynthioGold-onlyUnder review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

8 REPRODUCIBILITY STATEMENT

We provide our code in the supplementary material with this submission. All codes will be open-
sourced upon paper acceptance, including all T2A checkpoints. All experimental details, including
training parameters and hyper-parameters, are provided in Section 5.

REFERENCES

Haider Al-Tahan and Yalda Mohsenzadeh. Clar: Contrastive learning of auditory representations. In
International Conference on Artificial Intelligence and Statistics, pp. 2530–2538. PMLR, 2021.

Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet.
Synthetic data from diffusion models improves imagenet classification. Transactions on Machine
Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?
id=DlRsoxjyPm.

Max F Burg, Florian Wenzel, Dominik Zietlow, Max Horn, Osama Makansi, Francesco Locatello, and
Chris Russell. Image retrieval outperforms diffusion models on data augmentation. Transactions
on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/
forum?id=xflYdGZMpv.

Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang. An empirical survey of data
augmentation for limited data learning in nlp. Transactions of the Association for Computational
Linguistics, 11:191–211, 2021. URL https://api.semanticscholar.org/CorpusID:
235422524.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep
reinforcement learning from human preferences. Advances in neural information processing
systems, 30, 2017.

Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and
Alexandre D´efossez. Simple and controllable music generation. Advances in Neural Information
Processing Systems, 36, 2024.

Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset.
In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing
(ICASSP), pp. 736–740. IEEE, 2020.

Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning
audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE, 2023.

Zach Evans, CJ Carr, Josiah Taylor, Scott H Hawley, and Jordi Pons. Fast timing-conditioned latent

audio diffusion. arXiv preprint arXiv:2402.04825, 2024a.

Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio

open. arXiv preprint arXiv:2407.14358, 2024b.

Tiantian Feng, Dimitrios Dimitriadis, and Shrikanth Narayanan. Can synthetic audio from generative
foundation models assist audio recognition and speech modeling? arXiv preprint arXiv:2406.08800,
2024.

Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. FSD50K: an open
dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language
Processing, 30:829–852, 2022.

Jonas Geiping, Micah Goldblum, Gowthami Somepalli, Ravid Shwartz-Ziv, Tom Goldstein, and
Andrew Gordon Wilson. How much data are augmentations worth? an investigation into scaling
laws, invariance, and implicit regularization. arXiv preprint arXiv:2210.06441, 2022.

Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing
Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for
audio events. In 2017 IEEE international conference on acoustics, speech and signal processing
(ICASSP), pp. 776–780. IEEE, 2017.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio gener-
ation using instruction tuned llm and latent diffusion model. arXiv preprint arXiv:2304.13731,
2023.

Sreyan Ghosh, Ashish Seth, and S Umesh. Decorrelating feature spaces for learning general-purpose
audio representations. IEEE Journal of Selected Topics in Signal Processing, 16(6):1402–1414,
2022. doi: 10.1109/JSTSP.2022.3202093.

Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, S Ramaneswaran, S Sakshi, Utkarsh
Tyagi, and Dinesh Manocha. DALE: Generative data augmentation for low-resource legal NLP.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference
on Empirical Methods in Natural Language Processing, pp. 8511–8565, Singapore, December
2023a. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.528. URL
https://aclanthology.org/2023.emnlp-main.528.

Sreyan Ghosh, Ashish Seth, Srinivasan Umesh, and Dinesh Manocha. Mast: Multiscale audio
spectrogram transformers. In ICASSP 2023-2023 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), pp. 1–5. IEEE, 2023b.

Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi,
Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Gama: A large audio-language model
with advanced audio understanding and complex reasoning abilities, 2024a. URL https://
arxiv.org/abs/2406.11768.

Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Reddy Evuru, Ra-
maneswaran S, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Compa:
Addressing the gap in compositional reasoning in audio-language models. In The Twelfth Interna-
tional Conference on Learning Representations, 2024b. URL https://openreview.net/
forum?id=86NGO8qeWs.

Sreyan Ghosh, Utkarsh Tyagi, Sonal Kumar, Chandra Kiran Reddy Evuru, , Ramaneswaran S,
S Sakshi, and Dinesh Manocha. ABEX: Data augmentation for low-resource NLU via expanding
abstract descriptions. In The 62nd Annual Meeting of the Association for Computational Linguistics,
2024c.

Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. arXiv preprint

arXiv:2104.01778, 2021.

Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and XI-
AOJUAN QI. IS SYNTHETIC DATA FROM GENERATIVE MODELS READY FOR IMAGE
RECOGNITION? In The Eleventh International Conference on Learning Representations, 2023.
URL https://openreview.net/forum?id=nUmCcZ5RKF.

Calum Heggan, Sam Budgett, Timothy Hospedales, and Mehrdad Yaghoobi. Metaaudio: A few-shot
audio classification benchmark. In Artificial Neural Networks and Machine Learning – ICANN
2022, pp. 219–230, Cham, 2022. Springer International Publishing. ISBN 978-3-031-15919-0.

I Jordal. audiomentations, 2021. URL https://zenodo.org/record/13639627.

Akbar Karimi, Leonardo Rossi, and Andrea Prati. AEDA: An easier data augmentation technique
for text classification. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-
tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp.
2748–2754, Punta Cana, Dominican Republic, November 2021. Association for Computational
Linguistics. doi: 10.18653/v1/2021.findings-emnlp.234. URL https://aclanthology.
org/2021.findings-emnlp.234.

Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fr\’echet audio distance:
A metric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08466, 2018.

Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating
captions for audios in the wild. In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume
1 (Long and Short Papers), pp. 119–132, 2019.

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Jaeyeon Kim, Jaeyoon Jung, Jinjoo Lee, and Sang Hoon Woo. Enclap: Combining neural audio codec
and audio-text joint embedding for automated audio captioning. In ICASSP 2024 - 2024 IEEE
International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6735–6739,
2024. doi: 10.1109/ICASSP48485.2024.10446672.

Zhifeng Kong, Sang-gil Lee, Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, Rafael Valle,
Soujanya Poria, and Bryan Catanzaro. Improving text-to-audio models with synthetic captions.
arXiv preprint arXiv:2406.15487, 2024.

Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre D´efossez, Jade Copet,
Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation.
In The Eleventh International Conference on Learning Representations, 2023. URL https:
//openreview.net/forum?id=CYK7RfcOzQ4.

Cyril Laurier, Owen Meyers, Joan Serra, Martin Blech, Perfecto Herrera, and Xavier Serra. Indexing
music by mood: design and integration of an automatic content-based annotator. Multimedia Tools
and Applications, 48:161–184, 2010.

Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and
Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv
preprint arXiv:2301.12503, 2023.

Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang.
On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint
arXiv:2406.15126, 2024.

Vincent Lostanlen and Carmine-Emanuele Cella. Deep convolutional networks on the pitch spiral for
musical instrument recognition, 2017. URL https://arxiv.org/abs/1605.06644.

Navonil Majumder, Chia-Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, and Soujanya
Poria. Tango 2: Aligning diffusion-based text-to-audio generations through direct preference
optimization. arXiv preprint arXiv:2404.09956, 2024.

Pranay Manocha, Zeyu Jin, Richard Zhang, and Adam Finkelstein. Cdpam: Contrastive learning for
perceptual audio similarity. In ICASSP 2021-2021 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), pp. 196–200. IEEE, 2021.

Irene Mart´ın-Morat´o and Annamaria Mesaros. What is the ground truth? reliability of multi-annotator
data for audio tagging. In 2021 29th European Signal Processing Conference (EUSIPCO), pp.
76–80. IEEE, 2021.

Annamaria Mesaros, Toni Heittola, Aleksandr Diment, Benjamin Elizalde, Ankit Shah, Emmanuel
Vincent, Bhiksha Raj, and Tuomas Virtanen. Dcase 2017 challenge setup: Tasks, datasets and
baseline system. In DCASE 2017-Workshop on Detection and Classification of Acoustic Scenes
and Events, 2017.

Loris Nanni, Gianluca Maguolo, and Michelangelo Paci. Data augmentation approaches for improving

animal audio classification. Ecological Informatics, 57:101084, 2020.

An Thanh Nguyen, Byron Wallace, Junyi Jessy Li, Ani Nenkova, and Matthew Lease. Aggregating
and predicting sequence labels from crowd annotations. In Regina Barzilay and Min-Yen Kan (eds.),
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), pp. 299–309, Vancouver, Canada, July 2017. Association for Computational
Linguistics. doi: 10.18653/v1/P17-1028. URL https://aclanthology.org/P17-1028.

Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for
audio: Self-supervised learning for general-purpose audio representation. In 2021 International
Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE, 2021.

Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and
Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition.
arXiv preprint arXiv:1904.08779, 2019.

13

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Geoffroy Peeters. A large set of audio features for sound description (similarity and classification) in

the cuidado project. CUIDADO Ist Project Report, 54(0):1–25, 2004.

Karol J. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd
Annual ACM Conference on Multimedia, pp. 1015–1018. ACM Press. ISBN 978-1-4503-3459-
4. doi: 10.1145/2733373.2806390. URL http://dl.acm.org/citation.cfm?doid=
2733373.2806390.

Zafar Rafii, Antoine Liutkus, Fabian-Robert St¨oter, Stylianos Ioannis Mimilakis, and Rachel Bittner.
The MUSDB18 corpus for music separation, December 2017. URL https://doi.org/10.
5281/zenodo.1117372.

Zhao Ren, Kun Qian, Tanja Schultz, and Bj¨orn W. Schuller. An overview of the icassp special
session on ai security and privacy in speech and audio processing. In Proceedings of the 5th
ACM International Conference on Multimedia in Asia Workshops, MMAsia ’23 Workshops, New
York, NY, USA, 2023. Association for Computing Machinery.
ISBN 9798400703263. doi:
10.1145/3611380.3628563. URL https://doi.org/10.1145/3611380.3628563.

Julien Ricard. Towards computational morphological description of sound. DEA pre-thesis research

work, Universitat Pompeu Fabra, Barcelona, 2004.

Francesca Ronchini, Luca Comanducci, and Fabio Antonacci. Synthesizing soundscapes: Leveraging
text-to-audio models for environmental sound classification. arXiv preprint arXiv:2403.17864,
2024.

Aaqib Saeed, David Grangier, and Neil Zeghidour. Contrastive learning of general-purpose audio
representations. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and
Signal Processing (ICASSP), pp. 3875–3879. IEEE, 2021.

J. Salamon, C. Jacoby, and J. P. Bello. A dataset and taxonomy for urban sound research. In 22nd
ACM International Conference on Multimedia (ACM-MM’14), pp. 1041–1044, Orlando, FL, USA,
Nov. 2014.

Ashish Seth, Sreyan Ghosh, Srinivasan Umesh, and Dinesh Manocha. Slicer: Learning universal
audio representations using low-resource self-supervised pre-training. In ICASSP 2023-2023 IEEE
International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,
2023.

Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning.

Journal of big data, 6(1):1–48, 2019.

Janne Spijkervet. Spijkervet/torchaudio-augmentations, 2021. URL https://zenodo.org/

record/4748582.

Jinchuan Tian, Chunlei Zhang, Jiatong Shi, Hao Zhang, Jianwei Yu, Shinji Watanabe, and Dong Yu.
Preference alignment improves language model-based tts. arXiv preprint arXiv:2409.12403, 2024.

Brandon Trabucco, Kyle Doherty, Max A Gurinas, and Ruslan Salakhutdinov. Effective data
In The Twelfth International Conference on Learning

augmentation with diffusion models.
Representations, 2024. URL https://openreview.net/forum?id=ZWzUA9zeAg.

George Tzanetakis and Perry Cook. Multifeature audio segmentation for browsing and annotation.
In Proceedings of the 1999 IEEE Workshop on Applications of Signal Processing to Audio and
Acoustics. WASPAA’99 (Cat. No. 99TH8452), pp. 103–106. IEEE, 1999.

George Tzanetakis, Georg Essl, and Perry Cook. Automatic musical genre classification of audio

signals, 2001. URL http://ismir2001.ismir.net/pdf/tzanetakis.pdf.

Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam,
Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using
direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp. 8228–8238, 2024.

14

Under review as a conference paper at ICLR 2025

Jason Wang, Luis Perez, et al. The effectiveness of data augmentation in image classification using

deep learning. Convolutional Neural Networks Vis. Recognit, 11(2017):1–8, 2017.

Yu Wang, Nicholas J. Bryan, Mark Cartwright, Juan Pablo Bello, and Justin Salamon. Few-
In ICASSP 2021 - 2021 IEEE International
shot continual learning for audio classification.
Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 321–325, 2021. doi:
10.1109/ICASSP39728.2021.9413584.

Yuanyuan Wang, Hangting Chen, Dongchao Yang, Zhiyong Wu, Helen Meng, and Xixin Wu.
Audiocomposer: Towards fine-grained audio generation with natural language descriptions. arXiv
preprint arXiv:2409.12560, 2024.

Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text
classification tasks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings
of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th
International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6382–
6388, Hong Kong, China, November 2019. Association for Computational Linguistics. doi:
10.18653/v1/D19-1670. URL https://aclanthology.org/D19-1670.

Zeng Weili, Yichao Yan, Qi Zhu, Zhuo Chen, Pengzhi Chu, Weiming Zhao, and Xiaokang Yang.
Infusion: Preventing customized text-to-image diffusion from overfitting. In ACM Multimedia
2024, 2024.

Huang Xie and Tuomas Virtanen. Zero-shot audio classification via semantic embeddings. IEEE/ACM

Transactions on Audio, Speech, and Language Processing, 29:1233–1242, 2021.

Xuenan Xu, Zhiling Zhang, Zelin Zhou, Pingyue Zhang, Zeyu Xie, Mengyue Wu, and Kenny Q Zhu.
Blat: Bootstrapping language-audio pre-training based on audioset tag-guided synthetic data. In
Proceedings of the 31st ACM International Conference on Multimedia, pp. 2756–2764, 2023.

Yi Yuan, Dongya Jia, Xiaobin Zhuang, Yuanzhe Chen, Zhengxi Liu, Zhuo Chen, Yuping Wang,
Improving audio generation with visual

Yuxuan Wang, Xubo Liu, Mark D Plumbley, et al.
enhanced caption. arXiv preprint arXiv:2407.04416, 2024.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings
of the IEEE/CVF international conference on computer vision, pp. 6023–6032, 2019.

Shilei Zhang, Yong Qin, Kewei Sun, and Yonghua Lin. Few-shot audio classification with attentional

graph neural networks. In Interspeech, pp. 3649–3653, 2019.

Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning:
A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10795–10816,
2023.

Qihao Zhao, Yalun Dai, Hao Li, Wei Hu, Fan Zhang, and Jun Liu. Ltgc: Long-tail recognition
via leveraging llms-driven generated content. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR), pp. 19510–19520, June 2024.

A APPENDIX

Table of Contents:

• A.1 Background on Diffusion Models
• A.2 Prompts
• A.3 Examples
• A.4 Extra Results
• A.5 Dataset Details
• A.6 Algorithm

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

A.1 DIFFUSION MODELS

Diffusion models consist of two main processes: a forward process and a reverse process. Given
a data point x0 with probability distribution p(x0), the forward diffusion process gradually adds
Gaussian noise to x0 according to a pre-set variance schedule β1, · · · , βT and degrades the structure
of the data. At the time step t, the latent variable xt is only determined by the xt−1 due to its
discrete-time Markov process nature, and can be expressed as:

p(xt | xt−1) = N (xt; (cid:112)1 − βtxt−1, βtI),
(12)
As t increases over several diffusion steps, p(xT ) approaches a unit spherical Gaussian distribution.
The marginal distribution of xt at any given step can be expressed analytically as:
p(xt | x0) = N (xt;

(13)
where αt = (cid:81)t
s=1(1 − βs). The reverse process aims to reconstruct the original data from the
noise-corrupted version by learning a series of conditional distributions. The transition from xt to
xt−1 is modeled as:

αtx0, (1 − αt)I),

√

pθ(xt−1 | xt) = N (xt−1; µt−1

θ

, σt−1
θ

),

µt−1
θ =

(cid:18)

xt −

1
√
αt

βt√

1 − ¯αt

(cid:19)

ϵθ (xt, t)

,

(14)

(15)

1 − ¯αt−1
1 − ¯αt
i=1 αi, θ represents the learnable parameters, µt−1
is the mean estimate,
is the standard deviation estimate, and ϵθ(xt, t) is the noise estimated by the neural network.

where αt = 1 − βt, ¯αt = (cid:81)t
σt−12
θ
The reverse process estimates the data distribution p(x0) by integrating over all possible paths:

σt−12
θ

· βt,

(16)

=

θ

pθ(x0) =

(cid:90)

T
(cid:89)

pθ(xT )

pθ(xt−1 | xt) dx1 : T

(17)

t=1
where pθ(xT ) = N (xT ; 0, I). At inference time, the diffusion model iteratively executes the reverse
process (Eq. 17) T times starting from a randomly sampled Gaussian Noise (ϵ ∼ N (0, I)).

A.2 PROMPTS

Fig. 6, 7, 8 and 9 illustrate all the prompts used in our experiments. For all experiments, we prompt
GPT-4-Turbo (GPT-4-turbo-2024-04-09) with top-p=0.5 and temperature=0.7.

Figure 6: LLM prompt (Prompt 1) for extracting components from audio captions.

A.3 EXAMPLES

Table 6 presents examples of captions generated by the Synthio framework, along with their revised
versions for captions that were initially rejected.

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

I will provide you with a caption of an audio that describes the events taking place in theaudio. Additionally, I will also provide you with a label for the audio. Extract the phrasesthat correspond to the distinctive features of the audio. There are 3 types of features you needto extract:1) the unique foreground events in the caption,2) the broader background scene or background events in the or audio and3) any other features related to the audio. Return a JSON with key 3 keys, one as named as‘events’, the other as named as ‘scenes’, and the other named as ‘other features’, where thevalues of these keys correspond to a comma-separated pythonic list where each item in the listis a string corresponding to the extracted phrases. Please ignore any phrase that (exactly orsemantically) corresponds to the label of the audio. If you think there is no information foreither of the keys, leave them empty. Here is the caption:{}Here is the label:{}Under review as a conference paper at ICLR 2025

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Figure 7: LLM prompt (Prompt 2) for generating new audio captions given elements from existing captions.

Figure 8: LLM prompt for generating random captions for Random Captions baselines in Table 1.

A.4 EXTRA RESULTS

A.4.1 RESULTS ON THE FULL TRAINING SPLITS

Table 7 presents the performance comparison of Synthio on the full original dataset splits (where
the entire training set is used without any downsampling). While Synthio outperforms all baselines,
traditional augmentation methods prove to be much more competitive in this scenario. This contrasts
with the results in Table 1 where traditional augmentations showed minimal improvements in
performance.

Additional Discussion on Results. As we see in Table 1 (and Table 7), performance gains with
Synthio as the number of Gold samples increase (highest absolute gains with n = 100 and lowest
with full dataset). This phenomenon is consistent across prior work in vision (Trabucco et al., 2024),
text (Ghosh et al., 2023a; 2024c), and audio (Ronchini et al., 2024). Most synthetic data augmentation
methods demonstrate substantial gains in low-resource regimes, but these gains naturally diminish as
the quantity of high-quality labeled data increases (for example, Azizi et al. just show over ImageNet
only a modest improvement of just over 1%, where the authors reported when augmenting this
large-scale dataset).

17

I will provide you with a caption for an audio. The label generally describes the audio in anabstract fashion and mentions the broader scene or event that I need to teach an audio modelabout from the audio, i.e., the audio and its label is part of the training set for training anaudio classification model. I will also provide you with the domain of the audio which will helpyou identify the true sound conveyed in the label. I need you to rewrite the caption for meaccording to this set of rules:1. I will provide you with lists of various audio features corresponding to events, backgroundsor other features. You should rewrite the given caption such that it has has features inspiredfrom the features provided to you, i.e., you should try to describe a scene for the label with events, backgrounds and features similar but unique from the ones given.2. After re-writing, the caption should still obey the audio event label.Here is the label:{}. Here is the domain of the audio:{}.Here is the list of events:{}Here is the list of backgrounds:{}Here is the list of other features:{}Just output the rewritten caption and nothing else. Output 'None' if you did not rewrite.I will provide you with a label for an audio. The label generally describes the audio in anabstract fashion and mentions the broader scene or event that I need to teach an audio modelabout from the audio, i.e., the audio and its label is part of the training set for training anaudio classification model. I will also provide you with the domain of the audio which will helpyou identify the true sound conveyed in the label. I would like you to generate 5 new captionsthat describe the event or source in the label in diverse fashions. I will use these captions togenerate new audios that can augment my training set. Generate the new captions with thefollowing requirements:1. All the captions need to include new and diverse events and contexts beyond the actual eventconveyed by the label.2. Only add new events and context by understanding the broader context of the occurrence of theaudio and the target label. Do not add random events or contexts.3. The new caption should be not more than 20-25 words.4. However, after all these constraints and adding new events or contexts, the caption stillneeds to obey the event conveyed by the original label, i.e., the new caption may not lead to anaudio generation that defies the audio label.6. Finally, use the original label as a phrase in your caption.Here is the label:{}.Here is the domain of the audio:{}. Output a JSON with the key as the original label and a valueas the list of comma separated new captions. Only output the JSON and nothing elseUnder review as a conference paper at ICLR 2025

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Figure 9: LLM prompt (Prompt 3) for rewriting captions of rejected audios.

We hypothesize that this trend is rooted in the inherent diversity and richness of gold data. Gold
datasets typically capture nuanced variations and complex real-world distributions, including subtle
contextual and environmental factors that synthetic data struggles to replicate. Synthetic data, while
effective at filling gaps and addressing low-resource scenarios, often lacks the granularity necessary
to represent long-tail or edge-case instances. As the size of the gold dataset increases, the model
increasingly benefits from the inherent diversity of these high-quality examples, reducing the need
for synthetic data and its relative impact on performance.

Additionally, in Fig. 6 of their paper, Azizi et al. also how an increasing number of synthetic
augmentations leads to plateauing and even diminishing performance. We hypothesize that this is
due to over-fitting caused by lack of diversity in generated augmentations.

A.4.2 AUDIO GENERATION RESULTS FOR OUR TRAINED STABLE DIFFUSION

Table 9 presents a comparison of audio generation results across several evaluation metrics. We
evaluate our trained Stable Diffusion model (used in our experiments, including a version further
fine-tuned on AudioCaps) against other available models and baselines from the literature. Notably,
our model performs competitively with other fully open-source models across most metrics.

A.4.3 FAD SCORES FOR GENERATED AUGMENTATIONS

To offer an alternative perspective on the distributional consistency between the generated augmen-
tations and the ground-truth small-scale dataset, we compare the Fr´echet Audio Distance (FAD)
scores (Kilgour et al., 2018). For this experiment, we use Synthio with Template Captions. Table 10
presents a comparison of FAD scores between Synthio and other baselines. Synthio achieves the
highest FAD score, indicating that it produces the most consistent audio augmentations.

18

I will provide you with a label for an audio. The label generally describes the audio in anabstract fashion and mentions the broader scene or event that I need to teach an audio modelabout from the audio, i.e., the audio and its label is part of the training set for training anaudio classification model. I will also provide you with the domain of the audio which will helpyou identify the true sound conveyed in the label. I would like you to generate 5 new captionsthat describe the event or source in the label in diverse fashions. I will use these captions togenerate new audios that can augment my training set. Generate the new captions with thefollowing requirements:1. Each caption should have a diverse added events (beyond the event of the original label) andcontexts.2. Only add new events and context by understanding the broader context of the occurrence of theaudio and the target label. For adding events and contexts, please follow the next requirement.3. I will also provide you with a list of features extracted from an existing set of audios. Youshould try such that the new captions you generate for the label have a mix of events and scenessimilar to the events and background scenes that are given and new scenes, i.e., you should tryto describe a scene for the caption with the events and backgrounds provided to you in the givenlists but you should also add novel features (events, backgrounds or other features) beyond theones given.4. The new caption should be not more than 20-25 words.5. However, after all these constraints and adding new events or contexts, the caption stillneeds to obey the event label, i.e., the new caption may not lead to an audio generation thatdefies the audio label.6. Finally, use the original label as a phrase in your caption.Here is the label:{}.Here is the domain of the audio:{}.Here is the list of events:{}Here is the list of backgrounds:{}Here is the list of other features:{}Output a JSON with the key as the original caption and a value as the list of comma separatednew captions. Only output the JSON and nothing else.Under review as a conference paper at ICLR 2025

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Dataset

Label

USD8k

children playing

USD8k

children playing

USD8k

street music

USD8k

street music

TUT

TUT

TUT

TUT

airport

airport

bus

bus

NSynth

keyboard

NSynth

keyboard

NSynth

organ

NSynth

organ

Medley

Violin

Medley

Violin

Medley

Flute

Medley

Flute

AudioCaps -

AudioCaps -

Generated Caption

Revised Caption

Children playing in a bustling city park
with distant traffic noise
Children playing in a schoolyard during
recess with teacher’s whistle

Street music playing near a busy in-
tersection filled with honking cars and
pedestrians.
Street music from a bustling market as
people chatter and vendors shout

airport with people talking and walking
around in an empty hallway
In the airport, people are talking with
the sound of a crowd of people in the
background, as announcements play.

Bus passing by on a road while people
are chatting at a nearby cafe.
bus passing by on a road as it continues
to blow into the microphone

keyboard accompaniment to a live band
performance at a bustling cafe.
a man typing on a keyboard at office

A serene church service with an organ
playing a melody and soft brass are
playing.
An organ plays as guitars are playing
together in the background.

violin being played during a classical
symphony orchestra performance
violin performing a lively jig at a
bustling street fair

flute playing in a tranquil forest during
the early morning
Flute performance in a bustling city
park during a sunny afternoon.

A dog barks repeatedly in the back-
ground while a car engine starts
In the distance, a faint thunder rumble
is audible, accompanied by the gentle
rustling of leaves in the wind.

NA

Children playing in a neighborhood al-
ley with sound of distant construction

NA

Street music echoing through an alley-
way during a lively street festival.

NA

airport ambiance with people talking
and children running around

NA

bus idling on a road with birds chirping
nearby

NA

keyboard rhythms echoing in an empty
auditorium during a rehearsal break

NA

An organ plays during a lively music
festival with various instruments.

NA

Violin solo during a quiet candlelight
dinner in a fancy restaurant.

NA

Flute music echoing in an ancient stone
cathedral.

-

Soft rain falls on a metal roof, creating
a rhythmic tapping sound.

Table 6: Examples of generated and revised captions from the Synthio methodology.

Table 7: Comparison of Synthio and other baselines on the full original dataset splits (using all samples from
the original training set as Dsmall).

Method

USD8K GTZAN Medley VS

MSDB

Gold-only
Random Noise
Pitch Shift
Spec. Aug.
Audiomentations
Retrieval
Vanilla Syn. Aug.
Synthio (ours)

88.23
86.17
87.58
87.92
88.01
78.27
89.57
89.57

82.00
82.35
83.02
82.50
82.75
69.25
82.85
82.85

80.99
79.72
79.63
79.14
81.26
73.24
81.79
81.79

92.73
92.94
92.17
92.42
92.47
80.43
93.15
93.01

73.9
74.55
74.6
74.5
75.05
69.95
75.85
74.24

A.4.4 EFFECT OF CLAP FILTERING

In this section, we provide additional experiments to show the effect of CLAP filtering on the Synthio
pipeline. Table 11 compares the performance of Synthio with and without CLAP. As we can see,

19

Under review as a conference paper at ICLR 2025

Table 8: CLAP score between generated audios and the label.

n

Method

USD8K NSynth

Real
Vanilla Syn. Aug.
Synthio

100

w/ Template Captions
w/ ERM

Real
Vanilla Syn. Aug.
Synthio

200

w/ Template Captions
w/ ERM

12.67
14.34
31.26
29.31
24.15

10.13
12.55
21.87
20.31
17.14

14.46
17.54
27.32
26.62
21.54

9.4
12.91
16.16
15.82
13.04

Table 9: Comparison of our trained Stable Diffusion model on AudioCaps test set

Model

FAD PANN (↓)

FAD VGG (↓)

IS PANN (↑) CLAP LAION (↑)

AudioLDM2-large
Tango-Full0FT-AC
Tango 2
Make-an-Audio 2
Stable Audio VECaps (ours)
Stable Audio VECaps + AudioCaps-FT (ours)

32.50
18.47
17.19
11.75
15.12
14.93

1.89
2.19
2.54
1.80
2.21
2.19

8.55
8.80
11.04
-
15.07
15.42

0.45
0.57
0.52
0.60
0.57
0.56

Table 12 compares the performance of various values of p on 5 datasets and 2 values of n (500 and
100). As we see, higher or lower values of p do not affect the final performance significantly.

Our T2A model uses the same CLAP text encoder for generating audio. Consequently, most generated
audios are already highly aligned with the intended category label. However, the purpose of CLAP
filtering is to safeguard against cases where the LLM hallucinates and generates a caption that deviates
significantly from the intended label. In such cases, CLAP filtering ensures that audios generated
from hallucinated captions are discarded, preventing them from negatively impacting the learning
process.

A.4.5 EFFECT OF TRAINING DATA AND MODEL ARCHITECTURE FOR THE TEX-TO-AUDIO

MODEL

In this section, we train our T2A model using 1) a different model architecture (we replace Stable
Diffusion with Tango Ghosal et al. (2023)) different training data (we replaced Sound-VECaps with
AudioCaps). Table 13 compares thee results. As we can clearly see, while the model architecture
of the T2A model does not affect the performance, replacing the training data with a small and less
diverse dataset leads to significant drop in performance.

A.4.6 SYNTHIO AS A COMPLIMENTARY APPROACH TO TRADITIONAL AUGMENTATIONS

Table 14 compares results of Synthio augmentations when combined with traditional augmentations.
As we can see, Synthio boosts performance of all methods and combining traditional augmentations
with Synthio boosts Synthios overall performance. This shows that Synthio can act as a complimentary
step for traditional augmentations.

Additional Discussion. Across all datasets, we noticed that CLAP filtering removed at most 10%
of the generated samples. This confirms that the majority of the synthetic data is already well-
aligned with the target categories, and filtering primarily handles rare cases of misalignment. Thus
we emphasize on the point that while most generated audios align with the target label, the CLAP
filtering stage acts as a safeguard against hallucinations by the LLM, which may occasionally generate
captions that deviate significantly from the intended category. In such cases, filtering ensures that
misaligned audios are discarded, preventing them from negatively impacting model training.

20

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Under review as a conference paper at ICLR 2025

Table 10: Comparison of FAD score of Vaniall Syn. Aug. and Stable Audio VECaps (ours).

n

Dataset Model

FAD VGG (↓)

100 NSynth

200 TUT

Vanilla Syn. Aug.
Stable Audio VECaps (ours)

Vanilla Syn. Aug.
Stable Audio VECaps (ours)

1.83
1.42

1.71
1.45

Table 11: Ablation study evaluating the impact of CLAP filtering on Synthio’s performance.

n

50

100

200

500

Method

ESC-50 USD8K GTZAN TUT VS

Synthio
Synthio w/o CLAP

Syhtio
Synthio w/o CLAP

Syhtio
Synthio w/o CLAP

Syhtio
Synthio w/o CLAP

49.50
47.25

83.35
82.55

86.10
85.25

92.10
90.25

76.12
74.34

85.00
84.64

82.81
79.94

89.18
88.42

68.20
66.35

71.20
69.30

82.05
80.54

82.25
89.70

43.84
40.28

71.23
70.41

56.83
55.22

67.81
65.42

80.67
77.29

86.70
84.93

87.52
86.31

91.42
89.67

A.5 DATASET DETAILS

NSynth Instruments: NSynth is a large-scale dataset consisting of musical notes played by a variety
of instruments. It includes a rich set of acoustic features from instruments like guitars, flutes, and
more, providing diverse sound textures for classification tasks.

TUT Urban: The TUT Urban dataset captures everyday sounds from urban environments, including
noises like traffic, human activities, and construction.
It is commonly used for acoustic scene
classification and environmental sound recognition.

ESC-50: ESC-50 is a well-known dataset for environmental sound classification, containing 50
categories of everyday sounds such as animal noises, natural elements, and human activities, making
it suitable for multi-class classification challenges.

UrbanSound8K (USD8K): USD8K is a curated collection of urban sounds divided into ten classes,
including sirens, street music, and car horns. It is used widely for evaluating models on sound event
detection in real-world scenarios.

GTZAN: GTZAN is a music genre classification dataset that includes ten music genres such as pop,
rock, and jazz. It is a standard benchmark for evaluating music classification models, although it has
known data quality issues.

Medley-solos-DB: This dataset consists of solo recordings of different musical instruments, making it
valuable for studying isolated instrument sounds and training models for music instrument recognition.

MUSDB18: MUSDB18 is used primarily for music source separation tasks. It contains full-track
recordings of different music styles, providing a mix of vocals, drums, bass, and other instruments,
useful for multi-class classification.

DCASE Task 4: Part of the DCASE challenge, this dataset focuses on domestic sound scene and
event classification. It includes various audio clips recorded in home environments, often used for
anomaly detection and sound event classification.

Vocal Sounds (VS): This dataset includes various vocal sounds such as singing, speech, and vocal
effects, providing rich data for studying voice classification and enhancing models for vocal audio
recognition tasks.

21

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Under review as a conference paper at ICLR 2025

Table 12: Comparison of Synthio’s performance with different CLAP threshold levels.

n

p

ESC-50 USD8K GTZAN TUT

VS

50

100

200

500

0.85
0.3
0.5

0.85
0.3
0.5

0.85
0.3
0.5

0.85
0.3
0.5

49.50
47.10
48.25

83.35
82.55
82.70

86.10
85.25
85.70

92.10
90.25
91.65

76.12
74.14
75.39

85.00
84.64
84.73

82.81
79.94
80.30

89.18
88.42
89.07

68.20
67.50
67.75

71.20
69.30
70.25

82.05
80.55
81.30

82.25
80.70
81.05

43.84
41.17
41.93

71.23
70.41
70.86

56.83
55.22
56.19

67.81
65.42
66.35

80.67
79.32
79.48

86.70
84.93
85.22

87.52
86.31
87.11

91.42
89.67
90.02

Table 13: Comparison of Synthio with Synthio’s Stable Audio trained only wiht AudioCaps and Tango trained
with Sound-VECaps

n

50

100

Method

ESC-50 USD8K GTZAN Medley TUT

Synthio (ours)
Synthio w/ AudioCaps
Synthio w/ Tango

Synthio (ours)
Synthio w/ AudioCaps
Synthio w/ Tango

49.50
29.20
48.55

83.35
58.20
81.50

76.12
60.15
75.05

85.00
74.27
84.13

68.20
50.15
66.19

71.20
66.55
70.95

60.58
49.19
59.12

71.23
67.93
69.97

43.84
38.62
42.59

52.42
48.23
51.47

Table 14: Performance comparison of Synthio when paired with traditional augmentation techniques

n

Method

ESC-50 USD8K GTZAN Medley

50

100

Synthio (ours)
w/ Random Noise
w/ Pitch Shift
w/ Spec Aug
w/ Audiomentations

Synthio (ours)
w/ Random Noise
w/ Pitch Shift
w/ Spec Aug
w/ Audiomentations

49.50
49.65
49.80
50.95
50.35

83.35
83.85
83.60
84.25
84.10

76.12
77.31
78.52
77.93
77.24

85.00
86.59
86.32
86.17
85.95

68.20
70.15
69.50
70.35
69.50

71.20
71.60
72.95
72.75
72.85

60.58
61.54
60.29
61.17
61.53

71.23
72.35
72.50
73.05
72.87

A.6 ALGORITHM

Algorithm 1 algorithmically illustrated Synthio.

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

22

Under review as a conference paper at ICLR 2025

Algorithm 1 Synthio Framework for Audio Classification Augmentation

Require: Small human-annotated dataset Dsmall;

Noisy audio-caption paired dataset Da-c;
Number of generations per instance j;
Similarity threshold p%;
Maximum self-reflection iterations imax.

## Initial Training of T2A Model
Train T2A model T θ on Da-c.

## Construction of Preference Dataset Dpref
for each audio instance dk in Dsmall do

Create caption ck = “Sound of a labelk”.
for l = 1 to j do

Generate audio ˜ak,l = T θ(ck) starting from random noise.
Pair (˜ak,l, ak) where ak is the ground-truth audio.
Add pair to Dpref with ˜ak,l as loser and ak as winner.

end for

end for

## Preference Optimization Using DPO
Fine-tune T θ on Dpref using DPO methodology.

## Generating Diverse Prompts with MixCap
Use audio captioning model to generate captions for all ak in Dsmall.
Prompt LLM to extract acoustic components (backgrounds, events, their attributes and relations) from captions.

for each label labelk in Dsmall do

Using extracted acoustic elments, prompt LLM to generate n diverse captions {ck,1, ck,2, . . . , ck,n}.

end for

## Generation of Synthetic Data Dsyn
syn ← ∅, Drej
Initialize Dacc
for each caption ck,m do

syn ← ∅.

Generate audio ˜ak,m = T θ(ck,m).
Evaluate similarity sk,m = CLAP(˜ak,m, labelk).
if sk,m ≥ p% then

Add (˜ak,m, labelk) to Dacc
syn.

else

Add (ck,m, labelk) to Drej
syn.

end if
end for

## Self-Reflection and Caption Revision
Set iteration count i ← 0.
while Drej

syn ̸= ∅ and i < imax do

i ← i + 1.
for each rejected caption ck,m in Drej

syn do

k,m.

k,m = T θ(c′

Provide LLM with ck,m and insights from Dacc
syn.
Obtain revised caption c′
Generate audio ˜a′
Evaluate similarity s′
if s′
k,m ≥ p% then
Add (˜a′
Remove ck,m from Drej
syn.

k,m, labelk) to Dacc
syn.

k,m = CLAP(˜a′

k,m).

k,m, labelk).

Update ck,m ← c′

k,m in Drej
syn.

else

end if
end for
end while

## Final Training Dataset and Classification Model
Combine Dsyn with ground-truth data Dsmall to form Dtrain.
Train audio classification model on Dtrain.

23

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

