Regularizing Neural Networks with Meta-Learning Generative Models
This paper investigates methods for improving generative data augmentation
for deep learning. Generative data augmentation leverages the synthetic samples
produced by generative models as an additional dataset for classification with
small dataset settings. A key challenge of generative data augmentation is that
the synthetic data contain uninformative samples that degrade accuracy. This is
because the synthetic samples do not perfectly represent class categories in
real data and uniform sampling does not necessarily provide useful samples for
tasks. In this paper, we present a novel strategy for generative data
augmentation called meta generative regularization (MGR). To avoid the
degradation of generative data augmentation, MGR utilizes synthetic samples in
the regularization term for feature extractors instead of in the loss function,
e.g., cross-entropy. These synthetic samples are dynamically determined to
minimize the validation losses through meta-learning. We observed that MGR can
avoid the performance degradation of na\"ive generative data augmentation and
boost the baselines. Experiments on six datasets showed that MGR is effective
particularly when datasets are smaller and stably outperforms baselines.
Shin'ya Yamaguchi, Daiki Chijiwa, Sekitoshi Kanai, Atsutoshi Kumagai, Hisashi Kashima
# Regularizing Neural Networks

###### Abstract

This paper investigates methods for improving generative data augmentation for deep learning. Generative data augmentation leverages the synthetic samples produced by generative models as an additional dataset for classification with small dataset settings. A key challenge of generative data augmentation is that the synthetic data contain uninformative samples that degrade accuracy. This is because the synthetic samples do not perfectly represent class categories in real data and uniform sampling does not necessarily provide useful samples for tasks. In this paper, we present a novel strategy for generative data augmentation called _meta generative regularization_ (MGR). To avoid the degradation of generative data augmentation, MGR utilizes synthetic samples in the regularization term for feature extractors instead of in the loss function, e.g., cross-entropy. These synthetic samples are dynamically determined to minimize the validation losses through meta-learning. We observed that MGR can avoid the performance degradation of naive generative data augmentation and boost the baselines. Experiments on six datasets showed that MGR is effective particularly when datasets are smaller and stably outperforms baselines.

## 1 Introduction

While deep neural networks achieved impressive performance on various machine learning tasks, training them still requires a large amount of labeled training data in supervised learning. The labeled datasets are expensive when a few experts can annotate the data, e.g., medical imaging. In such scenarios, _generative data augmentation_ is a promising option for improving the performance of models. Generative data augmentation basically adds pairs of synthetic samples from conditional generative models and their target labels into real training datasets. The expectations of generative data augmentation are that the synthetic samples interpolate missing data points and perform as oversampling for classes with less real training samples [1]. This simple method can improve the performance of several tasks with less diversity of inputs such as medical imaging tasks [2; 3; 4; 5].

However, in general cases that require the recognition of more diverse inputs (e.g., CIFAR datasets [6]), generative data augmentation degrades rather than improves the test accuracy [7]. Previous studies have indicated that this can be caused by the low quality of synthetic samples in terms of the diversity and fidelity [7; 8]. If this hypothesis is correct, we can expect high-quality generative models (e.g., StyleGAN2-ADA [9]) to resolve the problem; existing generative data augmentation methods adopt earlier generative models e.g., ACGAN [10] and SNGAN [11]. Contrary to the expectation, this is not the case. We observed that generative data augmentation fails to improve models even when using a high-quality StyleGAN2-ADA (Figure 1). Although the samples partially appear to be real to humans, they are not yet sufficient to train classifiers in
existing generative data augmentation methods. This paper investigates methodologies for effectively extracting useful information from generative models to improve model performance.

We address this problem based on the following hypotheses. First, _synthetic samples are actually informative but do not perfectly represent class categories in real data_. This is based on a finding by Brock et al. [14] called "class leakage," where a class conditional synthetic sample contains attributes of other classes. For example, they observed failure samples including an image of "tennis ball" containing attributes of "dogs" (Figure 4(d) of [14]). These class leaked samples do not perfectly represent the class categories in the real dataset. If we use such class leaked samples for updating classifiers, the samples can distort the decision boundaries, as shown in Figure 3. Second, _regardless of the quality, the generative models originally contain uninformative samples to solve the tasks_. This is simply because the generative models are not explicitly optimized to generate informative samples for learning the conditional distribution \(p(y|x)\); they are optimized only for learning the data distribution \(p(x)\). To maximize the gain from synthetic samples, we should select appropriate samples for training tasks.

In this paper, we present a novel regularization method called _meta generative regularization_ (MGR). Based on the above hypotheses, MGR is composed of two techniques for improving generative data augmentation: _pseudo consistency regularization_ (PCR) and _meta pseudo sampling_ (MPS). PCR is a regularization term using synthetic samples in training objectives for classifiers. Instead of supervised learning with negative log-likelihood \(-\log p(y|x)\), i.e., cross-entropy, on synthetic samples, we regularize the feature extractor to avoid the distortions on decision boundaries. That is, PCR leverages synthetic samples only for learning feature spaces. PCR penalizes the feature extractors by minimizing the gap between variations of a synthetic sample, which is inspired by consistency regularization in semi-supervised learning [15; 16; 17]. MPS corresponds to the second hypothesis and its objective is to select useful samples for training tasks by dynamically searching optimal latent vectors of the generative models. Therefore, we formalize MPS as a bilevel optimization framework of a classifier and a finder that is a neural network for searching latent vectors. Specifically, this framework updates the finder through meta-learning to reduce the validation loss and then updates the classifier to reduce the PCR loss (Figure 2). By combining PCR and MPS, we can improve the performance even when the existing generative data augmentation degrades the performance (Figure 1).

We conducted experiments with multiple vision datasets and observed that MGR can stably improve baselines on various settings by up to 7 percentage points of test accuracy. Further, through the visualization studies, we confirmed that MGR utilizes the information in synthetic samples to learn feature representations through PCR and obtain meaningful samples through MPS.

## 2 Preliminary

### Problem Setting

We consider a classification problem in which we train a neural network model \(f_{\theta}:\mathcal{X}\rightarrow\mathcal{Y}\) on a labeled dataset \(\mathcal{D}=\{(x^{i},y^{i})\in\mathcal{X}\times\mathcal{Y}\}_{i=1}^{N}\), where \(\mathcal{X}\) and \(\mathcal{Y}\) are the input and output label spaces, respectively. Here, we can use a generative model \(G_{\Phi}:\mathcal{Z}\times\mathcal{Y}\rightarrow\mathcal{X}\), which is trained on \(\mathcal{D}\). We assume that \(G_{\Phi}\) generates samples from a latent vector \(z\in\mathcal{Z}\) and conditions on samples with a

Figure 1: Accuracy gain using meta generative regularization on Cars [12] with ResNet-18 classifier [13] and StyleGAN2-ADA [9] (FID: 9.5)
categorical label \(y\in\mathcal{Y}\), where \(z\) is sampled from a standard Gaussian distribution \(p(z)=\mathcal{N}(0,I)\) and \(y\) is uniformly sampled from \(\mathcal{Y}\)2. We refer to the classification task on \(\mathcal{D}\) as the main task, and \(f_{\theta}\) as the main model. \(f_{\theta}\) is defined by a composition of a feature extractor \(g_{\psi}\) and a classifier \(h_{\omega}\), i.e., \(f_{\theta}=h_{\omega}\circ g_{\psi}\) and \(\theta=[\psi,\omega]\). To validate \(f_{\theta}\), we can use a small validation dataset \(\mathcal{D}_{\text{val}}=\{(x_{\text{val}}^{i},y_{\text{val}}^{i})\in\mathcal{ X}\times\mathcal{Y}\}_{i=1}^{N_{\text{val}}}\), which has no intersection with \(\mathcal{D}\) (i.e., \(\mathcal{D}\cap\mathcal{D}_{\text{val}}=\emptyset\)).

Footnote 2: Although we basically use conditional \(G_{\Phi}\) for comparing our method and generative data augmentation, our method can be used with unconditional \(G_{\Phi}\) (see Sec. 4.6).

### Generative Data Augmentation

A typical generative data augmentation trains a main model \(f_{\theta}\) with both real data and synthetic data from the generative models [19]. We first generate synthetic samples to be utilized as additional training data for the main task. Most previous studies on generative data augmentation [19; 20; 8] adopt conditional generative models for \(G_{\Phi}\), and generate a pseudo dataset \(\mathcal{D}_{\text{p}}\) as

\[\mathcal{D}_{\text{p}}=\{(x_{\text{p}}^{i},y_{\text{p}}^{i})\mid x_{\text{p}}^ {i}=G_{\Phi}(z^{i},y_{\text{p}}^{i})\}_{i=1}^{N_{\text{p}}}, \tag{1}\]

where \(z^{i}\) is sampled from a prior distribution \(p(z)\), and \(y_{\text{p}}^{i}\) is uniformly sampled from \(\mathcal{Y}\). Subsequently, \(f_{\theta}\) is trained on both of \(\mathcal{D}\) and \(\mathcal{D}_{\text{p}}\) using the following objective function.

\[\min_{\theta} \mathcal{L}(\theta)+\lambda\mathcal{L}_{\text{p}}(\theta), \tag{2}\] \[\mathcal{L}(\theta) = \mathbb{E}_{(x,y)\in\mathcal{D}}\ell(f_{\theta}(x),y),\] (3) \[\mathcal{L}_{\text{p}}(\theta) = \mathbb{E}_{(x_{\text{p}},y_{\text{p}})\in\mathcal{D}_{\text{p}} }\ell_{\text{p}}(f_{\theta}(x_{\text{p}}),y_{\text{p}}), \tag{4}\]

where \(\ell\) is a loss function of the main task (e.g., cross-entropy), \(\ell_{\text{p}}\) is a loss function for the synthetic samples, and \(\lambda\) is a hyperparameter for balancing \(\mathcal{L}\) and \(\mathcal{L}_{\text{p}}\). In previous works, \(\ell_{\text{p}}\) is often set the same as \(\ell\). Although optimizing Eq. (2) with respect to \(\theta\) is expected to boost the test performance by interpolating or oversampling conditional samples [1], its naive application degrades the performance of \(f_{\theta}\) on general settings [7]. In this paper, we explore methods to resolve the degradation of generative data augmentation and maximize the performance gain from \(\mathcal{D}_{\text{p}}\).

## 3 Proposed Method

In this section, we describe our proposed method, MGR. The training using MGR is formalized as alternating optimization of a main model and finder network for searching latent vectors of \(G_{\Phi}\), as shown in Figure 2. To maximize the gain from synthetic samples, MGR regularizes a feature extractor \(g_{\psi}\) of \(f_{\theta}\) using PCR by effectively sampling useful samples for the generalization from \(G_{\Phi}\) using MPS. We show the overall algorithm of MGR in Appendix A.

Figure 4: Meta pseudo sampling framework. We meta-optimize the finder \(F_{\phi}\) to generate a useful latent vector \(F_{\phi}(z)\) for training a model parameter \(\theta\) through minimizing the validation loss with the once updated parameter \(\theta^{\prime}\) by a real sample \(x\) and a synthetic sample \(x_{\text{p}}=G_{\Phi}(F_{\phi}(z))\).

Figure 3: Distortion of decision boundary caused by generative data augmentation with conditional synthetic samples leaking another class attribute. If the synthetic sample \(x_{\text{p}}\) is “tennis ball dog” (reprinted from [18]) with its conditional label \(y_{\text{p}}=\) “ball”, the supervised learner of a task head \(h_{\omega}\) distorts the decision boundary of “dog” and “ball” to classify \(x_{\text{p}}\) as “ball”.
### Pseudo Consistency Regularization

As discussed in Secion 1, we hypothesize that the synthetic samples do not perfectly represent class categories, and training classifiers using them can distort the decision boundaries. This is because \(y_{\mathrm{p}}\) is not reliable due to \(\mathcal{D}_{\mathrm{p}}\) can contain class leaked samples [14]. To avoid the degradation caused by the distortion, we propose utilizing \(x_{\mathrm{p}}\) to regularize only the feature extractor \(g_{\phi}\) of \(f_{\theta}\) by discarding a conditional label \(y_{\mathrm{p}}\). For the regularization, we borrow the concept of consistency regularization, which was originally proposed for semi-supervised learning (SSL) [15; 16; 17]. These SSL methods were designed to minimize the dissimilarity between the two logits (i.e., the output of \(h_{\omega}\)) of strongly and weakly transformed unlabeled samples to obtain robust representations. By following this concept, the PCR loss is formalized as

\[\ell_{\text{PCR}}(x_{\mathrm{p}};\psi)=\|g_{\psi}(T(x_{\mathrm{p}}))-g_{\psi} (x_{\mathrm{p}})\|_{2}^{2}, \tag{5}\]

where \(T\) is a strong transformation such as RandAugment [21], which is similar to the one used in UDA [16] for SSL. The difference between PCR and UDA is that PCR penalizes only \(g_{\psi}\), whereas UDA trains the entire \(f_{\theta}=h_{\omega}\circ g_{\psi}\). \(\ell_{\text{PCR}}\) can be expected to help \(g_{\psi}\) learns features of inter-cluster interpolated by \(x_{\mathrm{p}}\) without distorting the decision boundaries. Using \(\ell_{\text{PCR}}\), we rewrite Eq. (4) as

\[\mathcal{L}_{\text{PCR}}(\theta)=\mathbb{E}_{x_{\mathrm{p}}\in\mathcal{D}_{ \mathrm{p}}}\ell_{\text{PCR}}(x_{\mathrm{p}};\psi). \tag{6}\]

### Meta Pseudo Sampling

Most generative data augmentation methods generate a synthetic sample \(x_{\mathrm{p}}\) from \(G_{\Phi}\) with a randomly sampled latent vector \(z\). This is not the best option as \(G_{\Phi}\) is not optimized for generating useful samples to train \(f_{\theta}\) in predicting the conditional distribution \(p(y|x)\); it is originally optimized to replicate \(p(x|y)\) or \(p(x)\). The main concept of MPS is to directly determine useful samples for training \(f_{\theta}\) by optimizing an additional neural network called a _finder_\(F_{\phi}:\mathcal{Z}\rightarrow\mathcal{Z}\). \(F_{\phi}\) takes a latent vector \(z\sim p(z)\) as input and outputs a vector of the same dimension as \(z\). By using \(F_{\phi}\), we generate a synthetic sample as \(x_{\mathrm{p}}=G_{\Phi}(F_{\phi}(z),y)\). The role of \(F_{\phi}\) is to find the optimal latent vectors that improve the training of \(f_{\theta}\) through \(x_{\mathrm{p}}\). Although we can consider optimizing \(G_{\Phi}\) instead of \(F_{\phi}\), we observed that optimizing \(G_{\Phi}\) is unstable and causes the performance degradations of \(f_{\theta}\) (Section 4.5).

The useful samples for generalization should reduce the validation loss of \(f_{\theta}\) by using them for optimization. Based on this simple suggestion, we formalize the following bilevel optimization problem for \(F_{\phi}\).

\[\min_{\phi}\mathcal{L}_{\text{val}}(\theta^{*})=\mathbb{E}_{(x_{ \text{val}},y_{\text{val}})\in\mathcal{D}_{\text{val}}}\ell(f_{\theta^{*}}(x_{ \text{val}}),y_{\text{val}})\] \[\text{subject to}\qquad\quad\theta^{*}=\operatorname*{argmin}_{ \theta}\mathcal{L}(\theta)+\lambda\mathcal{L}_{\text{PCR}}(\theta,\phi). \tag{7}\]

Note that the finder parameter \(\phi\) is added to the arguments of \(\mathcal{L}_{\text{PCR}}\). We can optimize \(F_{\phi}\) with a standard gradient descent algorithm because \(F_{\phi}\), \(G_{\Phi}\), and \(f_{\theta}\) are all composed of differentiable functions as well as existing meta-learning methods such as MAML [22] and DARTS [23]. We approximate \(\theta^{*}\) because the exact computation of the gradient \(\nabla_{\phi}\mathcal{L}_{\text{val}}(\theta^{*})\) is expensive [22; 23]:

\[\theta^{*}\approx\theta^{\prime}=\theta-\eta\nabla_{\theta}(\mathcal{L}( \theta)+\lambda\mathcal{L}_{\text{PCR}}(\theta,\phi)), \tag{8}\]

where \(\eta\) is an inner step size. Thus, we update \(F_{\phi}\) by using \(\theta^{\prime}\) that is updated for a single step and then alternately update \(\theta\) by applying Eq. (6) with \(x_{\mathrm{p}}\) generated from the updated \(F_{\phi}\). The overall optimization flow is shown in Figure 4.

Approximating gradients with respect to \(\phi\).Computing \(\nabla_{\phi}\mathcal{L}_{\text{val}}(\theta^{\prime})\) requires the product computation including second-order gradients: \(\nabla_{\phi,\theta}^{2}\mathcal{L}_{\text{PCR}}(\theta,\phi)\nabla_{\theta^{ \prime}}\mathcal{L}_{\text{val}}(\theta^{\prime})\). This causes a computation complexity of \(\mathcal{O}((|\Phi|+|\phi|)|\theta|)\). To avoid this computation, we approximate the term using the finite difference method [23] as

\[\nabla_{\phi,\theta}^{2}\mathcal{L}_{\text{PCR}}(\theta,\phi)\nabla_{\theta^{ \prime}}\mathcal{L}_{\text{val}}(\theta^{\prime})\approx\frac{\nabla_{\phi} \mathcal{L}_{\text{PCR}}(\theta^{+},\phi)-\nabla_{\phi}\mathcal{L}_{\text{PCR}}( \theta^{-},\phi)}{2\varepsilon}, \tag{9}\]

where \(\theta^{\pm}\) is \(\theta\) updated by \(\theta\pm\eta\nabla_{\theta}\mathcal{L}_{\text{val}}(\theta)\). \(\varepsilon\) is defined by \(\frac{\text{const}}{\|\nabla_{\theta}\mathcal{L}_{\text{val}}(\theta)\|_{2}}\). We used \(0.01\) of the constant for \(\varepsilon\) based on [23]. This approximation reduces the computation complexity to \(\mathcal{O}(|\Phi|+|\phi|+|\theta|)\). We confirm the speedup by Eq. (9) in Appendix B.1.
Techniques for improving finderWe introduce two techniques for improving the training of \(F_{\phi}\) in terms of the architectures and a penalty term for the outputs.

While arbitrary neural architectures can be used for the implementation of \(F_{\phi}\), we observed that the following residual architecture produces better results.

\[F_{\phi}(z):=z+\tanh(\text{MLP}_{\phi}(z)), \tag{10}\]

where MLP is multi layer perceptron.

To ensure that \(F_{\phi}(z)\) does not diverge too far from the distribution \(p(z)\), we add a Kullback-Leibler divergence term \(D_{\text{KL}}(p_{\phi}(z)\|p(z))\), where \(p_{\phi}(z)\) is the distribution of \(F_{\phi}(z)\) into Eq. (7). When \(p(z)\) follows the standard Gaussian \(\mathcal{N}(0,I)\), \(D_{\text{KL}}(p_{\phi}(z)\|p(z))\) can be computed by

\[D_{\text{KL}}(p_{\phi}(z)\|p(z))=-\frac{1}{2}(1+\log\sigma_{\phi}-\mu_{\phi}^{ 2}-\sigma_{\phi}), \tag{11}\]

where \(\mu_{\phi}\) and \(\sigma_{\phi}\) is the mean and variance of \(\{F_{\phi}(z^{i})\}_{i=1}^{N_{\text{p}}}\). In Appendix B.2, we discuss the effects of design choice based on the ablation study.

## 4 Experiments

In this section, we evaluate our MGR (the combination of PCR and MPS with the experiments on multiple image classification datasets. We mainly aim to answer three research questions with the experiments: (1) Can PCR avoid the negative effects of \(x_{\text{p}}\) in existing methods and improve the performance of \(f_{\theta}\)? (2) Can MPS find better samples for training than that by uniform sampling? (3) How practical is the performance of MGR? We compare MGR with baselines including conventional generative data augmentation and its variants in terms of test performance (Sec. 4.2 and 4.3). Furthermore, we conduct a comprehensive analysis of PCR and MPS such as the visualization of trained feature spaces (Sec. 4.4), quantitative/qualitative evaluations of the synthetic samples (Sec. 4.5), performance studies when changing generative models (Sec. 4.6), and comparison to data augmentation methods such as TrivialAugment [24] (Sec. 4.7).

### Settings

Baselines.We compare our method with the following baselines. **Base Model**: training \(f_{\theta}\) with only \(\mathcal{D}\). **Generative Data Augmentation (GDA)**: training \(f_{\theta}\) with \(\mathcal{D}\) and \(G_{\Phi}\) using Eq. (2). **GDA+MH**: training \(f_{\theta}\) with \(\mathcal{D}\) and \(G_{\Phi}\) by decoupling the heads into \(h_{\omega}\) for \(\mathcal{D}\) and \(h_{\omega_{\text{p}}}\) for \(\mathcal{D}_{\text{p}}\). MH denotes multi-head. This is a naive approach to avoid the negative effect of \(x_{\text{p}}\) on \(h_{\omega}\) by not passing \(x_{\text{p}}\) through \(h_{\omega}\). GDA+MH optimizes the parameters as \(\operatorname*{argmin}\limits_{\theta}\mathcal{L}(f_{\theta}(x),y)+\lambda \mathcal{L}(h_{\omega_{\text{p}}}(g_{\psi}(x_{\text{p}})),y_{\text{p}})\).

**GDA+SSL**: training \(f_{\theta}\) with \(\mathcal{D}\) and \(G_{\Phi}\) by applying an SSL loss for \(\mathcal{D}_{\text{p}}\) that utilizes the output of \(h_{\omega}\) unlike PCR. This method was originally proposed by Yamaguchi et al. [25] for transfer learning, but we note that \(G_{\Phi}\) was trained on the main task dataset \(\mathcal{D}\) in contrast to the original paper. By following [25], we used UDA [16] as the SSL loss and the same strong transformation \(T\) as of MGR for the consistency regularization.

Datasets.We used six image datasets for classification tasks in various domains: Cars [12], Aircraft [26], Birds [27], DTD [28], Flowers [29], and Pets [30]. Furthermore, to evaluate smaller dataset cases, we used subsets of Cars that were reduced by \(\{10,25,50,75\}\%\) in volume; we reduced them by random sampling on a fixed random seed. We randomly split a dataset into \(9:1\) and used the former as \(\mathcal{D}\) and the latter as \(\mathcal{D}_{\text{val}}\).

Architectures.We used ResNet-18 [13] as \(f_{\theta}\) and generators of conditional StyleGAN2-ADA for \(256\times 256\) images [18] as \(G_{\Phi}\). \(F_{\phi}\) was composed of a three-layer perceptron with a leaky-ReLU activation function. We used the ImageNet pre-trained weights of ResNet-18 distributed by PyTorch.3 For StyleGAN2-ADA, we did not use pre-trained weights. We trained \(G_{\Phi}\) on each \(\mathcal{D}\) from scratch according to the default setting of the implementation of StyleGAN2-ADA.4 Note that we used the same \(G_{\Phi}\) in the baselines and our proposed method.

Footnote 3: [https://github.com/pytorch/vision](https://github.com/pytorch/vision) 4: [https://github.com/NVlabs/stylegan2-ada](https://github.com/NVlabs/stylegan2-ada)
Training.We trained \(f_{\theta}\) by the Nesterov momentum SGD for 200 epochs with a momentum of 0.9, and an initial learning rate of 0.01; we decayed the learning rate by 0.1 at 60, 120, and 160 epochs. We trained \(F_{\phi}\) by the Adam optimizer for 200 epochs with a learning rate of \(1.0\times 10^{-4}\). We used mini-batch sizes of 64 for \(\mathcal{D}\) and 64 for \(\mathcal{D}_{\mathrm{p}}\). The input samples were resized into a resolution of \(224\times 224\); \(x_{p}\) was resized by differentiable transformations. For synthetic samples from \(G_{\Phi}\) in PCR and GDA+SSL, the strong transformation \(T\) was RandAugment [21] by following [16], and it was implemented with differentiable transformations provided in Kornia [31]. We determined the hyperparameter \(\lambda\) by grid search among \([0.1,1.0]\) with a step size of \(0.1\) for each method by \(\mathcal{D}_{\mathrm{val}}\). To avoid the overfitting, we set the hyperparameters of MGR that are searched with only applying PCR i.e., we did not use meta-learning to choose them. We used \(\lambda_{\text{KL}}\) of \(0.01\). We selected the final model by checking the validation accuracy for each epoch. We ran the experiments three times on a 24-core Intel Xeon CPU with an NVIDIA A100 GPU with 40GB VRAM and recorded average test accuracies with standard deviations evaluated on the final models.

### Evaluation on Multiple Datasets

We confirm the efficacy of MGR across multiple datasets. Table 0(a) shows the top-1 accuracy scores of each method. As reported in [7], GDA degraded the base model on many datasets; it slightly improved the base model on only one dataset. GDA+MH, which had decoupled classifier heads for GDA, exhibited a similar trend to GDA. This indicates that simply decoupling the classifier heads is not a solution to the performance degradation caused by synthetic samples. In contrast, our MGR stably and significantly outperformed the baselines and achieved the best results. The ablation of MGR discarding PCR or MPS is listed in Table 0(a). We confirm that both PCR and GDA+MPS improve GDA. While GDA+SSL underperforms the base models, PCR outperforms the base models. This indicates that using unsupervised loss alone is not sufficient to eliminate the negative effects of the synthetic samples and that discarding the classifier \(h_{\omega}\) from the regularization is important to obtain the positive effect. MPS yields only a small performance gain when combined with GDA, but it significantly improves its performance when combined with PCR i.e., MGR. This suggests that there is no room for performance improvements in GDA, and MPS can maximize the potential benefits of PCR.

### Evaluation on Small Datasets

A small dataset setting is one of the main motivations for utilizing generative data augmentation. We evaluate the effectiveness of MGR on smaller datasets. Table 0(b) shows the performance when reducing the Cars dataset into a volume of \(\{10,25,50,75\}\%\). Note that we trained \(G_{\Phi}\) on each reduced dataset, not on \(100\%\) of Cars. In contrast to the cases of the full dataset (Table 0(a)), no baseline methods outperformed the base model in this setting. This is because \(G_{\Phi}\) trained on the small datasets generates low-quality samples with less reliability on the conditional label \(y_{\mathrm{p}}\) that are not appropriate in supervised learning. On the other hand, MGR improved the baselines in large margins. This indicates that, even when the synthetic samples are not sufficient to represent the class categories, our MGR can maximize the information obtained from the samples by utilizing them to regularize feature extractors and dynamically finding useful samples.

### Visualization of Feature Spaces

In this section, we discuss the effects of PCR and MPS through the visualizations of feature spaces in training. To visualize the output of \(g_{\psi}\) in 2D maps, we utilized UMAP [32] to reduce the dimensions.

\begin{table}

\end{table}
Table 1: Top-1 accuracy (%) of ResNet18. Underlined scores outperform that of Base Model, and **Bolded scores** are the best among the methods.
UMAP is a visualization method based on the structure of distances between samples, and the low dimensional embeddings can preserve the distance between samples of the high dimensional input. Thus, the distance among the feature clusters on UMAP visualization of \(g_{\psi}\) can represent the difficulty of the separation by \(h_{\omega}\). We used the official implementation by [32]5 and its default hyperparameters. We plotted the UMAP embeddings of \(g_{\psi}(x)\) and \(g_{\psi}(x_{\mathrm{p}})\) at \(\{5,15,30\}\) epochs, as shown in Figure 5; we used ResNet-18 trained on the Cars dataset. At first glance, we observe that GDA and PCR formed completely different feature spaces. GDA forms the feature spaces by forcing to treat the synthetic samples the same as the real samples through cross-entropy loss and trying to separate the clusters of samples according to the class labels. However, the synthetic samples leaked to the inter-cluster region at every epoch because they could not represent class categories perfectly as discussed in Sec. 1. This means that the feature extractor might be distorted to produce features that confuse the classifier. On the other hand, the synthetic samples in PCR progressively formed a cluster at the center, and the outer clusters can be seen well separated. Since UMAP can preserve the distances between clusters, we can say that the sparse clusters that exist far from the center are considered easy to classify, while the dense clusters close to the center are considered difficult to classify. In this perspective, PCR helps \(g_{\psi}\) to leverage the synthetic samples for learning feature representations interpolating the dense difficult clusters. This is because the synthetic samples tend to be in the middle of clusters due to their less representativeness of class categories. That is, PCR can utilize the partial but useful information contained in the synthetic samples while avoiding the negative effect. Further, we observe that applying MPS accelerates the convergence of the synthetic samples into the center. From these observations, we conclude that PCR and MPS can help models learn useful feature representations for solving tasks.

Footnote 5: [https://github.com/lmcinnes/umap](https://github.com/lmcinnes/umap)

### Analysis of MPS

Evaluation of validation loss.We investigate the effects on validation losses when using MPS. Through the meta-optimization by Eq. (7), MPS can generate samples that reduce the validation loss of \(f_{\theta}\). We recorded the validation loss per epoch when applying uniform sampling (Uniform) and when applying MPS. We used the models trained on Cars and applied the PCR loss on both models. Figure 5(a) plots the averaged validation losses. MPS reduced the validation loss. In particular, MPS were more effective in early training epochs. This is related to accelerations of converging the central cluster of synthetic samples discussed in Section 4.4 and Figure 5. That is, MPS can produce effective samples for regularizing features and thus speed up the entire training of \(f_{\theta}\).

Quantitative evaluation of synthetic samples.We evaluate the synthetic samples generated by MPS. To assess the characteristics of the samples, we measured the difference between the data

Figure 5: UMAP visualization of feature spaces on training. The plots in the figures represent real and synthetic samples in the feature spaces. Our methods (PCR and PCR+MPS) can help the feature extractors separate real sample clusters. In contrast, the existing method (GDA) confuses the feature extractor by leaking synthetic samples out of the clusters.
distribution and distribution of the synthetic samples. We leveraged the Frechet Inception distance (FID, [33]), which is a measurement of the distribution gap between two datasets using the closed-form computation assuming multivariate normal distributions:

\[\text{FID}(\mathcal{D},\mathcal{D}_{\text{p}})=\|\mu-\mu_{\text{p}}\|_{2}^{2}+ \operatorname{Tr}\left(\Sigma+\Sigma_{\text{p}}-2\sqrt{\Sigma\Sigma_{\text{p}}} \right),\]

where \(\mu\) and \(\Sigma\) are the mean and covariance of the feature vectors on InceptionNet for input \(\{x^{i}\}\). Since FID is a distance, the lower \(\text{FID}(\mathcal{D},\mathcal{D}_{\text{p}})\) means that \(\mathcal{D}_{\text{p}}\) contains more high-quality samples in terms of realness. We computed FID scores using 2048 samples in \(\mathcal{D}\) and 2048 synthetic samples every epoch; the other settings are given in Section 4.1. The FID scores in training are plotted in Figure 5(b). We confirm that MPS consistently produced higher-quality samples than Uniform. This indicates that the sample quality is important for generalizing \(f_{\theta}\) even in PCR, and uniform sampling can miss higher quality samples in generative models. Since the performance gain by GDA+MPS in Table 0(a) did not better than MGR, the higher-quality samples by MPS can still contain uninformative samples for the cross-entropy loss, but they are helpful for PCR learning good feature representations.

Qualitative evaluation of synthetic samples.We evaluate the qualitative properties of the synthetic samples by visualizing samples of a class. The samples generated by Uniform and MPS are shown in Figure 6(a) and 6(b), where the dataset was Cars and the class category was Hummer. Compared with Uniform, MPS produced samples with more diverse body colors and backgrounds. That is, MPS focuses on the color and background of the car as visual features for solving classifications. In fact, since Hummer has various colors of car bodies and can drive on any road with four-wheel drive, these selective generations of MPS are considered reasonable in solving classification tasks.

### Effect of Generative Models

Here, we evaluate MGR by varying the generative model \(G_{\Phi}\) for producing synthetic samples. As discussed in Sec. 3.1 and 3.2, MGR can use arbitrary unconditional/conditional generative models as \(G_{\Phi}\) unless it has a latent space. To confirm the effects when changing \(G_{\Phi}\), we tested MGR with FastGAN [34], BigGAN [18], and StyleGAN-XL [35]. Table 2 shows the results on Cars. The unconditional FastGAN achieved similar improvements as conditional cases. However, since unconditional generative models are generally of low quality in FID, they are slightly less effective. For the conditional generative models, we observe that MGR performance improvement increases as the quality of synthetic samples improves. These results suggest the potential for even greater MGR gains in the future as the performance of the generative model improves. We also evaluate MGR with recent diffusion models in Appendix B.3.

### Combination of MGR and Data Augmentation

To assess the practicality of MGR, we evaluate the comparison and combination of MGR and existing data augmentation methods. Data augmentation (DA) is a method applied to real data and is an independent research field from generative data augmentation. Therefore, MGR can be combined with DA to improve performance further. Table 3 shows the evaluation results when comparing and combining MGR and DA methods; we used AugMix [36], RandAugment [21], and TrivialAugment [24] as the DA methods. The improvement effect of MGR is comparable to that of DA, and the highest accuracy was achieved by combining the two methods. This suggests that the proposed method using synthetic samples and DA plays different roles in classifier training and that MGR is also very effective in practical use.
## 5 Related Work

We briefly review generative data augmentation and training techniques using generative models.

The earliest works of generative data augmentation are [19; 37; 38]. They have demonstrated that simply adding synthetic samples as augmented data for classification tasks can improve performance in few-shot learning, person re-identification, and medical imaging tasks. Tran et al.[20] have proposed a generative data augmentation method that simultaneously trains GANs and classifiers for optimizing \(\theta\) to maximize the posterior \(p(\theta|x)\) by an EM algorithm. Although this concept is similar to our MPS in terms of updating both \(G_{\Phi}\) and \(f_{\theta}\), it requires training the specialized neural architectures based on GANs. In contrast, MPS is formalized for arbitrary existing generative models with latent variables, and it requires no restrictions to the training objectives of generative models.

On the analysis of generative data augmentation, Shmelkov et al. [7] have pointed out that leveraging synthetic samples as augmented data degrades the performance in general visual classification tasks. They have hypothesized that the cause of the degradation is the less diversity and fidelity of synthetic samples from generative models. Subsequent research by Yamaguchi et al. [8] have shown that the scores related to the diversity and fidelity of synthetic samples (i.e., SSIM and FID) are correlated to the test accuracies when applying the samples for generative data augmentation in classification tasks. Based on these works, our work reconsiders the training objective and sampling method in generative data augmentation and proposes PCR and MPS.

More recently, He et al. [39] have reported that a text-to-image generative model pre-trained on massive external datasets can achieve high performance on few-shot learning tasks. They also found that the benefit of synthetic data decreases as the amount of real training data increases, which they attributed to a domain gap between real and synthetic samples. In contrast, our method does not depend on any external dataset and successfully improves the accuracy of the classifier even when synthetic data are not representatives of class labels (i.e., having domain gaps).

## 6 Limitation

One of the limitations of our method is the requirement of bilevel optimization of classifier and finder networks. This optimization is computationally expensive in particular when used with generative models that require multiple inference steps, such as diffusion models as discussed in Appendix B.3. We have tried other objective functions not requiring bilevel optimization, but at this time, we have not found an optimization method that outperforms MPS (see Appendix B.4). Nevertheless, since recent studies rapidly and intensively focus on the speedup of diffusion models [40; 41; 42], we can expect that this limitation will be negligible in near the future.