colorlinks=true, linkcolor=blue, filecolor=blue, urlcolor=blue,
citecolor=blue,

mode=titleNextConvGeN

1]Institute of Computer Science, University of Rostock; Germany 2]Leibniz-
Institute for Food Systems Biology; Technical University of Munich, Freising;
Germany 3]Stellenbosch Institute for Advanced Study, South Africa 4]Indian
Institute of Science Education and Research Thiruvananthapuram, India

[ca1]Manjunath Mahendra is the first author of this publication. Chaithra
Umesh as the second author contributed significantly with the experiemnts.
Kristian Schultz helped with computational aspects of the project. The
conceptualization of the project was done by Saptarshi Bej. Saptarshi Bej and
Olaf Wolkenhauer together supervised the manuscript writing and the
experiments.
Corresponding author: Saptarshi Bej<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Convex space learning for tabular synthetic data generation

Manjunath Mahendra  Chaithra Umesh  Saptarshi Bej  Kristian Schultz  Olaf
Wolkenhauer  [  [  [  [

###### Abstract

Generating synthetic samples from the convex space of the minority class is a
popular oversampling approach for imbalanced classification problems.
Recently, deep-learning approaches have been successfully applied to modeling
the convex space of minority samples. Beyond oversampling, learning the convex
space of neighborhoods in training data has not been used to generate entire
tabular datasets. In this paper, we introduce a deep learning architecture
(NextConvGeN) with a generator and discriminator component that can generate
synthetic samples by learning to model the convex space of tabular data. The
generator takes data neighborhoods as input and creates synthetic samples
within the convex space of that neighborhood. Thereafter, the discriminator
tries to classify these synthetic samples against a randomly sampled batch of
data from the rest of the data space. We compared our proposed model with five
state-of-the-art tabular generative models across ten publicly available
datasets from the biomedical domain. Our analysis reveals that synthetic
samples generated by NextConvGeN can better preserve classification and
clustering performance across real and synthetic data than other synthetic
data generation models. Synthetic data generation by deep learning of the
convex space produces high scores for popular utility measures. We further
compared how diverse synthetic data generation strategies perform in the
privacy-utility spectrum and produced critical arguments on the necessity of
high utility models. Our research on deep learning of the convex space of
tabular data opens up opportunities in clinical research, machine learning
model development, decision support systems, and clinical data sharing.

###### keywords:

Tabular synthetic data, Convex space learning, GAN, privacy, and utility

##  1 Introduction

We previously introduced models that utilize convex space learning to overcome
imbalanced classification problem [1, 2, 3]. In these models, depending on the
data distribution, synthetic samples are generated from specific regions of
the minority class’s convex space, customized using linear interpolation on
original samples. A convex space is the convex hull of the data space. The
data points within a convex hull of a small neighborhood in the data space are
assumed to share similar statistical characteristics in the convex space
learning philosophy [2]. The idea of convex space learning is to generate
synthetic samples from appropriate regions of the convex space of the original
data. The appropriate regions depend on the dataset and the task at hand, and
such regions are ‘learned.’ For learning, traditional oversampling approaches
use predefined rules that specify which regions of the convex space are
appropriate for synthetic sample generation. However, these models are
currently limited to skewed or imbalanced datasets with a clear distinction
between majority and minority classes.

Recently, a convex space learning-based deep neural network model developed to
tackle the imbalance issue in tabular data is the ConvGeN model by Schultz et
al. [3]published on Patter Recognition journal. This model merges the convex
space learning technique with deep generative modeling to create synthetic
samples that balance the target classes for tabular data. The oversampling
results with ConvGeN have been significant, at par with the state-of-the-art
oversampling methods for imbalanced classification. However, this model was
not generalized for the complete table generation. Following upon this, we
broadened the application of the ConvGeN model beyond just imbalanced
learning. We have adapted the model in a more general setting, exploring its
potential for generating entire tabular data with diverse feature types.

Note that oversampling for imbalanced datasets is nothing but synthetic data
generation. In the context of synthetic data generation, data modalities such
as images or text are more popular. Significant progress has been made in
synthetic data generation for computer vision and Natural Language Processing
(NLP), offering exciting possibilities for clinical research. Among the most
commonly used models in computer vision are Variational Auto Encoders (VAEs)
[4, 5], Generative Adversarial Networks (GANs) [6, 7], and Denoising Diffusion
Probabilistic Models (DDPM) [8].

Recently, synthetic data generation models have also been developed for
tabular data, a widely used format for organizing and analyzing information
consisting of rows and columns. Each column in a table represents a specific
feature or attribute of the samples in rows [9, 10]. With well-defined
features and an interpretable structure, clinical tabular data leads to more
interpretable models.

However, in certain domains, such as bio-medicine, tabular data are subject to
privacy. Therefore, access to data with confidential patient information,
which includes sensitive medical records and personal identifiers, is
difficult [11, 12]. Such data must be protected to meet the requirements of
privacy laws such as the General Data Protection Regulation (GDPR) in the
European Union [13]. Moreover, even when access is granted, the scale of
available clinical data may only sometimes meet the requirements of modern
data-driven approaches.

Tabular synthetic data generation strategies are recently gaining popularity
to counter challenges related to data privacy and data enrichment [4, 14, 15,
16]. The idea is to generate synthetic data that mimics the characteristics of
real-world clinical data. If the synthetic data is shared instead of the real
data, then ideally, there is no breach of privacy. However, there is an
important point to note here. If the synthetic data is too similar to the real
data (in terms of statistical and other empirical measures), then an
‘attacker’ (an entity interested in privacy breach) can track the original
data back from the synthetic data. This defeats the purpose of synthetic data
generation. On the other hand, if the synthetic data is too dissimilar to the
original data, then any model built on the synthetic data might produce
inconsistent results, reducing the utility of such data. A synthetic data
generation model should preserve privacy and utility in an ideal scenario.
There has been no objective study that determines how well tabular synthetic
data generation models balance privacy and utility measures.

In this paper, we contribute to the ongoing research on synthetic tabular data
generation at the following two points:

  1. 1.

We generalize the existing ConvGeN algorithm such that it can be used for
synthetic tabular data generation in general, thereby proposing the
NextConvGeN algorithm. The model comprises a generator and discriminator
network, which cooperate among themselves (instead of competing like in
adversarial learning) to learn regions in the convex data space that are more
apt for sampling synthetic data points. We demonstrate the model’s
effectiveness in comparison with existing tabular synthetic data generation
algorithms using publicly available benchmarking datasets from the clinical
domain.

  2. 2.

We provide an objective comparison of major tabular synthetic data generation
models in terms of how they balance between privacy and utility based
performance measures.

In addition to these, the article further informs the community about some
more points of interest, such as:

  1. 1.

Most of the synthetic tabular data generation models currently available focus
on larger datasets [14, 15, 16]. However, in the clinical domain, one
frequently encounters smaller datasets. Our research reveals the effectiveness
of several tabular synthetic data generation strategies on such datasets.

  2. 2.

One-hot encoding is used for existing generative models to handle categorical
features with numerous categories. However, this makes the original dataset
sparse, which might affect synthetic data generation. The NextConvGeN model we
proposed in this article addresses this issue.

Our research paves the way for a novel tabular synthetic data generation
modeling approach. Moreover, it provides the community with a guideline on
which synthetic data generation to choose depending on the purpose (whether
preserving utility is more important or preserving privacy is more important).

##  2 Related research

VAEs and GANs are two pioneering techniques in generative modeling that have
significantly shaped research on synthetic data. Initially used for synthetic
image generation, now they have found diverse applications across various
domains. This section discusses the origins and advancements of VAEs and GANs,
highlighting their contributions to synthetic data generation [17]. Then, we
discuss specialized approaches designed to tackle challenges posed by tabular
data.

###  2.1 Variational AutoEncoders (VAEs)

VAE uses deep learning and probabilistic modeling to generate synthetic data.
They have an encoder and a decoder and introduce a probabilistic
interpretation of the latent space to account for underlying variability. A
reparameterization method is used during sampling, and the training process
maximizes the lower bound of evidence [4, 17, 18]. VAEs have advantages for
data augmentation and creating more comprehensive datasets but also have
limitations, such as difficulty handling discrete data. They can be used for
anomaly detection but may suffer from information loss, posterior collapse,
and sensitivity to prior distribution, likelihood function, and regularization
term [17].

###  2.2 Generative Adversarial Networks (GANs)

GANs revolutionized synthetic data generation by providing a robust framework
for creating highly realistic and diverse samples. It was proposed by
Goodfellow et al. in 2014 [19]. GANs have two primary components: a generator
network that produces artificial samples from random noise inputs during
training and a discriminator network that distinguishes between real and
generated samples. Once trained, GANs can produce synthetic samples with
complex patterns, textures, and structures, making them suitable for various
applications like image synthesis, text generation, and data augmentation.
GANs have several challenges that make them difficult to train. These include
oscillatory loss, uninformative loss, mode collapse, hyperparameter tuning,
and vanishing gradients. These problems are common to GANs regardless of
domain or architecture [17, 6, 7]. Addressing these issues is crucial for
successful GAN training and high-quality data generation. Since the
introduction of GANs, the field has seen significant advancements, including
Conditional GAN, Deep Convolutional GAN, Coupled GAN, Auxiliary Classifier
GAN, Wasserstein GANs, and more [7].

###  2.3 Conditional Generative Adversarial Network (CTGAN)

In 2019, Xu et al. presented a new method for synthesizing data in tabular
format using CTGAN. This method aims to model both continuous and categorical
columns effectively. The technique involves a unique normalization approach
for continuous features and a strategy to handle imbalances in discrete data.
For continuous features, CTGAN employs a three-step process. Initially, it
uses a variational Gaussian mixture model to model non-Gaussian and multimodal
continuous distributions and estimate the number of modes. Each continuous
value is then represented as a one-hot vector indicating the mode and a scalar
indicating the value within that mode. Categorical columns, on the other hand,
are represented through one-hot encoding. The final representation of a row in
the tabular data is obtained by concatenating continuous and categorical
columns [14].

To address the challenge of imbalanced discrete data, CTGAN introduces a
conditional vector, a one-hot-encoded vector representing the selection of a
specific value for a categorical column. During training, a column with
discrete values is uniformly selected, and a specific value is sampled based
on the probability mass function across all possible values. The generator is
then fed with this conditional vector and a noise vector, and the loss
function is modified to penalize the generator for failing to produce the
selected column with the chosen value. The concept of ”packing” is introduced
to prevent mode collapse in the generator. Training CTGAN involves providing a
list of discrete columns; others are treated as continuous data [14].
According to the results of the experiments, CTGAN outperformed other
generative models, including CLBN, PrivBN, MedGAN, VeeGAN, and TableGAN,
regarding machine learning efficacy on real datasets [14].

###  2.4 Conditional TABle GAN (CTAB-GAN)

Imbalanced data classes and skewed distributions are prevalent problems in
datasets across different fields [2, 3, 15]. These challenges can hinder
effective data analysis, research, and practical applications. To overcome
these, researchers have devised an approach to tackle this issue through CTAB-
GAN, the extension of CTGAN. This method combines conditional generation and
sampling-based training techniques to enhance the representation of all
classes, particularly those that are underrepresented, and preprocess
variables with long-tailed distributions using a logarithmic transformation.

Introducing a ”conditional vector” to the generator allows it to adapt its
synthesis procedure based on specific conditions, thereby improving the
representation of all classes. The logarithmic transformation aids in
compressing and reducing the distance between tail and bulk data, facilitating
the encoding of all values, including tail ones, by the variational Gaussian
mixture. To ensure the quality of synthetic data, researchers have introduced
two terms into the generator’s loss function- information loss and
classification loss. These terms minimize statistical differences between
generated and real data, encourage the generator to match the real dataset’s
statistical characteristics, and predict labels for synthetic data instances,
ensuring adherence to expected semantic criteria. CTAB-GAN addresses
challenges with continuous variables by employing mode-specific normalization,
resulting in more accurate synthesis [15].

Comparative evaluation against four other tabular data generators has
demonstrated CTAB-GAN’s high utility, statistical similarity to real data, and
reasonable safeguards for sensitive information. However, it is essential to
note that GAN-based models, including CTAB-GAN, require a large sample size
for training. Therefore, users should consider this aspect when assessing the
practical applicability of CTAB-GAN, particularly in scenarios where resource
constraints may pose challenges. CTAB-GAN is a promising technique that offers
a potential solution to a class imbalance in datasets, enabling practical data
analysis, research, and applications [15].

###  2.5 CTAB-GAN+

In addition to the research gaps addressed in CTGAN and CTAB-GAN, Zhao et al.
states that existing GAN-based tabular data generators overlook variables like
Mortgage in the Loan dataset, which can be numerical and categorical. This
oversight generates inaccuracies such as approximating zero values instead of
exact zeros and assigning meaning to negative values where there is no
reality. They have utilized sampling and mode-specific normalization methods
from CTGAN to deal with imbalanced data and multimodal distribution,
respectively. Additionally, to enhance the stability of GAN training, CTAB-
GAN+ has incorporated the Wasserstein plus gradient penalty for the loss
function, which was developed to address the exploding and vanishing gradient
concerns that can arise in the original WGAN [20]. Furthermore, to safeguard
the privacy of contributors while enabling meaningful aggregate analysis, the
encoder part of the model uses differential privacy, which introduces
controlled noise into datasets, ensuring that individual data points remain
indistinguishable within the statistical analysis. CTAB-GAN+ outperforms other
generative models regarding machine utility and statistical similarity, making
it an excellent choice for data generation [21].

###  2.6 Tabular Denoising Diffusion Probabilistic Model (TabDDPM)

TabDDPM uses diffusion models, known for generating images, and adjusts them
to generate tabular data. These models add noise to the input data through
iterative diffusion and then reverse it to produce a synthetic sample close to
the original distribution. Recently, Kotelnikov et al. introduced TabDDPM, a
state-of-the-art method generative model for handling diverse features in
tabular data [16]. They address differences in feature types by preprocessing
continuous features with Gaussian quantile transformation and representing
categorical features with one hot encoding. The model uses multinomial
diffusion for categorical features and Gaussian diffusion for continuous
features. Each categorical feature undergoes a separate diffusion process,
where noise components are sampled independently for all features. A
multilayer perceptron models the reverse diffusion step, which takes noisy
input data and predicts Gaussian and multinomial diffusion outcomes. Training
involves minimizing mean squared error for Gaussian diffusion and KL
divergence for multinomial diffusion. A class conditional model is used for
classification, and regression, the target is treated as an additional
numerical feature. TabDDPM performs well on benchmark datasets, consistently
generating high-quality synthetic samples compared to existing GAN or VAE-
based models [16]. However, this model treats categorical features with one-
hot encoding, which makes the original data sparse. This might affect the
synthetic data generation, especially when the sample size is small. In
addition, the TabDDPM model processes the feature types separately, possibly
not capturing the relationship between different feature types.

##  3 Methodology

This section introduces the NextConvGeN model, designed to generate synthetic
clinical tabular data using convex space learning. We will provide a brief
overview of NextConvGeN and the modifications made to the ConvGeN model to
create it. Additionally, we will delve into the mathematical foundation and
operational mechanisms of NextConvGeN.

###  3.1 Overview of NextConvGeN model

NextConvGeN model is developed with its underlying principle based on the
ConvGeN model. The learning approach of NextConvGeN is based on deep
cooperative learning, which involves using two neural networks to learn convex
coefficients during the training process. The generator network produces
synthetic samples within the convex space of the input data neighborhood,
while the discriminator network classifies these samples against the rest of
the data space. The networks work together, learning from each other’s
feedback and improving their capabilities over multiple iterations.
NextConvGeN learns the optimal coefficients for convex combinations during
training by adjusting the neural network’s parameters iteratively.

Unlike ConvGeN, which takes shuffled batches of minority class data
neighborhoods as inputs, NextConvGeN’s generator is designed to receive
shuffled neighborhood batches from the entire data space as its inputs. These
neighborhoods are identified using a unique nonlinear dimensional reduction
approach called Feature-type Distributed Clustering (FDC) introduced by Bej et
al. [22, 23]. This approach provides a more effective framework for generating
informative low-dimensional embedding from high-dimensional tabular data than
conventional Uniform Manifold Approximation and Projection (UMAP), a commonly
used nonlinear dimension reduction technique with a choice of similarity
measures. FDC applies the UMAP technique independently to each feature type
(continuous, ordinal, and nominal) using specific similarity measures [22].
The resulting lower dimension is more meaningful, considering the diverse
feature types present in the data. Hence, we explore the neighborhoods based
on this lower dimension.

A training input for the NextConvGeN model contains data points from

  1. (i)

a specific data neighborhood randomly shuffled

  2. (ii)

a shuffled batch of data points sampled from the original data, equal in size
to the considered data neighborhood, and disjoint to the chosen data
neighborhood.

where (i), as described above, is passed on to the generator, which generates
synthetic samples from the neighborhood. Those synthetic samples are
classified against (ii) by the discriminator. By cooperating with the
discriminator, the generator learns the appropriate coefficients of convex
combinations for synthetic data generation in a way that improves the
discriminator’s classification performance.

Note that the ConvGeN model could only be used in an imbalanced classification
where there are well-defined minority and majority classes. Our proposed model
does not require any such labels and can be used for general synthetic tabular
data generation. Anticipating the primary application of the proposed model in
the clinical and biomedical domain, an important change that we incorporated
in the novel model is the choice of neighborhoods by incorporating different
distance measures for various feature types (continuous, ordinal, and
nominal).

###  3.2 Mathematical interpretation of NextConvGeN model

For a given data point $x\in X$, where $X$ is a tabular dataset, we can denote
the $k$-neighborhood of $x\in X$ uniquely accessed by accounting for diverse
feature types as $N(x)\subseteq X$. We assume that $N(x)$ (in short, we will
sometimes call this $N$) has been randomly shuffled. The Generator $G$ will
take as input $N(x)$ and map it to the convex space of $N(x)$, which we can
call $\mathbb{C}_{N(x)}$:

$G:N(x)\subseteq X\rightarrow C_{N}(x)\subseteq\mathbb{C}_{N(x)},$ (1)

where $C_{N}(x)$ (we will refer to this as $C_{N}$ alternatively) is a convex
combination of the samples in $N(x)$. Note that here, we impose a constraint
on $G$ generating synthetic samples from the convex space of the original data
neighborhood. This ensures that the distribution of generated synthetic
samples, if considered as random variables from the corresponding data
neighborhood, has an identical mean to that of the original samples in that
neighborhood. A mathematical proof of this is provided by Bej et al. [2].

The discriminator $D$ receives as input the synthetic samples generated from a
particular neighborhood $N(x)$ by $G$ and learns to classify them against a
randomly shuffled batch of samples $M\subset N^{c}(x)$, such that $\lvert
M\rvert=\lvert C_{N}\rvert$ and $s\in M\iff s\notin N(x)$

$D:C_{N}\oplus M\subset N^{c}(x)\rightarrow[0,1],$

where $\oplus$ denotes a concatenation and $N^{c}(x)$, is the complement of
$N(x)$. $D$ thus ensures that the synthetic samples generated as convex
combinations of samples in $N(x)$ are distinct from the rest of the data
space. $D$ can pass on its knowledge to $G$ during the interactive training
process, such that $G$ learns to choose appropriate convex coefficients for
synthetic sample generation within $N(x)$. Learning the proper convex
coefficients by $G$ can be intuitively interpreted as if the model makes sure
the distribution of generated samples, in addition to having an identical mean
as the distribution of the original samples, also has an optimal variance that
assists the synthetic data generation even when a small amount of training
data is available.

We used the binary cross-entropy loss to train the $D$, which is defined as:

$L_{D}=-\sum_{c\in C_{N},y\in
M}\log\big{(}D(c)\big{)}+\log\big{(}1-D(y)\big{)}$ (2)

There is no separate training for the generator, as we assume that the convex
combinations of input neighborhood data points generated by $G$ are already
similar to the original samples (distributions share identical means). The
weights of $G$ are updated when the whole model is trained. The model uses the
Mean-Squared Error (MSE) as a loss function:

$L_{\text{Final}}=\text{MSE}(Y_{N\oplus M},\hat{Y}_{N\oplus M})$ (3)

where $Y_{N\oplus M}$ are the labels of the concatenated neighborhood batch
(shuffled) $N$ and the random batch $M$ (shuffled), while $\hat{Y}_{N\oplus
M}$ are the corresponding predicted labels.

Figure 1: Architecture of the NextConvGeN model. The generator component takes
shuffled data neighborhood batches as input and produces a convex combination
of the samples from the neighborhood. The discriminator component is
responsible for classifying these generated samples and comparing them against
a randomly selected outside neighborhood batch of the same size as the
synthetic inside neighborhood. The discriminator can be further trained and
repurposed as the Discriminator as a classifier after completing the
NextConvGeN model training.

###  3.3 Architectural design and operational principles of NextConvGeN

The NextConvGeN model is depicted in Figure 1. Its architecture is similar to
ConvGeN. The neural networks that form components of the NextConvGeN model
have a generic shallow structure, keeping in mind that we are not optimizing
the model for specific applicative purposes but rather performing benchmarking
studies. In NextConvGeN, the input layer of the generator takes neb nodes.
Each of them contains $f$ features. Where neb is the size of the neighborhood,
and $f$ is the number of features. $G$ receives a shuffled batch of neb
original samples during each iteration. The unique aspect is that each input
batch represents a neighborhood $N$ accessed through a dimension reduction
technique, utilizing suitable similarity measures for the feature types. This
neighborhood $N$ consists of the neb closest data points to a specific
original data point. Consequently, the dimension of the input becomes
$\texttt{neb}\times f$.

A 1-D convolution layer is employed to reduce the input dimension following
the input layer. Subsequently, the output of the convolution layer is
flattened and reshaped in a way to obtain a matrix denoted as ${K}_{N}$, whose
columns contain convex coefficients. In other words, each column in this
matrix sums up to one, and the entries are non-negative. The matrix
${K}_{N}^{T}$ represents the transpose of $K_{N}$, ensuring that each row of
${K}_{N}^{T}$ sums up to one. To further refine ${K}_{N}^{T}$, an operation
called Alpha-clipping is applied. It is a unique technique that involves
subtracting a certain percentage (alpha_clip) from the element with the
maximum value in each row of ${K}_{N}^{T}$ and adding this clipped amount to
the minimum value of the respective row. This process ensures that none of the
elements in each row equals one. This again ensures that a synthetic sample
generated using the convex combinations of the original samples from the data
neighborhood is not too similar to the original samples themselves.

Next, a residual connection is established between the input neighborhood $N$
and the matrix $C_{N}$. The matrix $C_{N}$ is obtained by multiplying
${K}_{N}^{T}$ with $N$. Each row of $C_{N}$ represents a convex combination of
the real data points within the neighborhood $N$. This means that the values
in each row of $C_{N}$ are calculated as a weighted average of the
corresponding real data points, resulting in a combination that reflects the
properties of the local neighborhood.

The input layer of $D$ consists of $f$ nodes, representing the number of
features. During each training step, $D$ receives a batch of synthetic samples
$C_{N}$ generated by $G$, concatenated with a batch of real data samples $M$
randomly chosen from $N^{c}$ and shuffled. $D$ incorporates three dense layers
with sizes $250$, $125$, and $75$, respectively. The output layer of $D$
consists of two nodes corresponding to the two classes (Synthetic samples from
a Neighborhood $N$ vs Random samples chosen from anywhere in the Data but the
corresponding Neighborhood $N$) being considered. A pseudo-code for the
NextConvGeN algorithm is shown in Algorithm 1

Inputs :
data Labeled data with minority and majority class.

Parameters : disc_train_count | $\geq 0$ |  Extra training steps for the discriminator before updating generator once. (used value: $5$)
---|---|---
neb | $\geq 2$ | Size of the minority neighborhood used to model convex minority data space.
gen | $\geq\texttt{neb}$ |  Number of generated synthetic points by generator at a time. (used value: $\texttt{gen}=\texttt{neb}$)
fdc | $...$ | $...$

Function train(data)

begin

$\texttt{normalized\\_data}\leftarrow fdc(data)$

$\texttt{neighbourhoods}\leftarrow$ Build neighbourhoods with neb points size
for all $x\in\texttt{data}$ with closeness according to normalized_data.

$\texttt{labels}\leftarrow\big{(}\underbrace{(1,0),...,(1,0)}_{gen},\underbrace{(0,1),...,(0,1)}_{gen}\big{)}$

For $1\ldots\texttt{neb\\_epochs}$ do

For $1\ldots$ disc_train_count do

For $x\in$ data do

min_batch $\leftarrow$ Set of neb nearest points to x in randomized order
according to neighborhoods.

maj_batch $\leftarrow$ List of gen random points from the dataset without the
points in min_batch.

conv_samples $\leftarrow$ Generate gen points using the neighborhood of x.

corrected_samples
$\leftarrow\texttt{correct\\_feature\\_types}(\texttt{min\\_batch},\texttt{conv\\_samples})$

concat_sample
$\leftarrow(\texttt{corrected\\_samples}\oplus\texttt{maj\\_batch})$

train discriminator with concat_sample $\mapsto$ labels

endfor

endfor

For $x\in$ minority class do

min_batch $\leftarrow$ Set of neb nearest points to x in randomized order.

maj_batch $\leftarrow$ List of gen random points from the dataset without the
points in min_batch.

concat_sample
$\leftarrow(\texttt{generator}(\texttt{min\\_batch})~{}\oplus~{}\texttt{maj\\_batch})$

train generator by using $\texttt{discriminator}(\texttt{concat\\_sample})$
with concat_sample $\mapsto$ labels

endfor

endfor

end

Algorithm 1 NextConvGeN train algorithm

Parameters of FDC and NextConvGeN algorithms The algorithms discussed in this
section, the FDC and the NextConvGeN algorithm, employ specific parameters to
configure their behavior and customize their operations. To facilitate the
reproducibility of our work, here we specify some details of our experiments.

FDC algorithm parameters: The FDC algorithm requires three main parameters to
create an instance of the library. These parameters are cont_list, nom_list,
and ord_list.

  1. 1.

cont_list: It represents the list of continuous feature column names. In the
context of the FDC algorithm, continuous features refer to variables with
numerical values that can take any real number.

  2. 2.

nom_list: This parameter represents the list of nominal feature column names.
Nominal features represent categorical variables with discrete values with no
particular order or hierarchy.

  3. 3.

ord_list: This parameter is used to specify the list of ordinal feature column
names. Ordinal features are similar to nominal ones but have a specific order
or hierarchy.

It should be noted that all other parameters of the FDC algorithm have default
values predefined within the library, i.e., they have predetermined values
unless otherwise provided.

NextConvGeN algorithm parameters: The NextConvGeN algorithm utilizes three
parameters: neb, gen, and disc_train_count.

  1. 1.

neb: Determines the number of neighboring data points for generating synthetic
points in the convex space.

  2. 2.

gen: This parameter determines the number of generated synthetic points the
generator creates simultaneously. These synthetic points are generated through
convex combinations of the real samples chosen from a neighborhood.

  3. 3.

disc_train_count: It indicates the number of additional training steps for the
discriminator before updating the generator once. The discriminator is a
component of the algorithm that helps assess the quality and authenticity of
the generated synthetic points.

  4. 4.

fdc: Requires the class instance of FDC to be passed, as it is responsible for
searching the neighborhoods that serve as input for the Generator.

  5. 5.

alpha_clip: This takes up a value between $0$ and, $1$ representing the
percentage of clipping the maximum convex coefficient.

We have set the gen value equal to neb, to generate synthetic points that
match the size of the input neighborhood.

Additionally, the term epoch is commonly used in conventional deep learning to
denote one training step where the entire training data is used once. However,
in the case of NextConvGeN, the term neighborhood epoch is used. This means
that during a training step, $G$ receives all real data neighborhoods as one
input batch, each time concatenated with a randomly shuffled data batch of
equal size. Each neighborhood (denoted as $N(x)$) in the input batch is passed
through $G$ once. Lastly, our algorithms and experiments discussed are
available here: https://github.com/manjunath-mahendra/NextConvGeNGit.

##  4 Datasets and evaluation measures

###  4.1 Datasets used and general protocol

In our benchmarking experiments, we used a diverse set of ten publicly
available datasets. These datasets vary regarding the number of samples they
contain and the characteristics they represent. A detailed overview of each
dataset is available in Table 1, providing insights into various attributes
that highlight the diversity of the datasets.

We performed the necessary preprocessing steps to ensure that the datasets
were free of missing values. Additionally, we have confirmed that all
variables are numerical rather than strings, as some generative models cannot
handle non-numeric variables. Also, it is essential to specify the different
feature types (continuous, ordinal and nominal) for the generative models.
Next, we partitioned each dataset into two parts: $70\%$ of the data was used
to train the generative model; whereas the remaining $30\%$ was taken as
holdout data for evaluation, allowing us to test the model’s performance on
previously unseen data. To determine the effectiveness of the proposed
NextConvGeN model, we compared it against adopting four state-of-the-art
approaches as benchmarks: TVAE, CTGAN, CTAB-GAN, CTAB-GAN+, and TabDDPM. We
generate synthetic samples five times the size of real training samples from
each generative model. Later, we down-sample (to the size of the training
data) to maintain the cardinality of the target feature as in the real
training data. This down-sampled synthetic data is then used for both utility
and privacy evaluation. By downsampling to match the cardinality (number of
unique values) of the target feature in the real data, we create a fair
comparison where all the models are evaluated on the datasets with the same
class distribution as real.

Dataset | Train size | Test size | Features | Ordinal | Nominal | Continuous
---|---|---|---|---|---|---
https://www.kaggle.com/datasets/andrewmvd/heart-failure-clinical-dataHeart failure | $209$ | $90$ | $12$ | $5$ | $0$ | $7$
https://doi.org/10.24432/C52P4XHeart disease | $211$ | $91$ | $13$ | $4$ | $4$ | $5$
https://www.kaggle.com/datasets/mysarahmadbhat/lung-cancerLung cancer | $216$ | $93$ | $15$ | $5$ | $9$ | $1$
https://codeocean.com/capsule/1269964/tree/v1/data/migraine.csvMigraine | $263$ | $114$ | $23$ | $6$ | $16$ | $1$
https://www.kaggle.com/datasets/fedesoriano/cirrhosis-prediction-dataset/codeLiver cirrhosis | $292$ | $126$ | $18$ | $4$ | $3$ | $11$
https://doi.org/10.24432/C5D02CIndian liver patients | $396$ | $170$ | $10$ | $0$ | $1$ | $9$
https://www.kaggle.com/datasets/uciml/pima-indians-diabetes-databasePima Indian diabetes | $537$ | $231$ | $8$ | $1$ | $0$ | $7$
https://doi.org/10.24432/C59W2DContraceptive methods | $997$ | $428$ | $9$ | $4$ | $4$ | $1$
https://doi.org/10.24432/C5H31ZObesity | $1460$ | $627$ | $16$ | $9$ | $4$ | $3$
https://www.kaggle.com/datasets/fedesoriano/stroke-prediction-dataset/codeStroke | $3435$ | $1473$ | $10$ | $3$ | $4$ | $3$
Table 1: Description of the datasets used in our benchmarking studies. This
table provides information on the dataset train and test sizes, the number of
attributes, and the distribution of ordinal, nominal, and continuous features
within each dataset.

###  4.2 Evaluation measures

Evaluation measures are crucial in determining the effectiveness and
reliability of synthetic data generated to represent real data. However, there
are no standard measures for evaluating tabular synthetic data [24]; hence, we
have chosen some commonly used measures from the available sources for
benchmarking. To compare the quality of synthetic data generated by several
models, we evaluate the synthetic data in terms of Utility and Privacy.

Utility measures for synthetic data are determined by the performance of
synthetic data for any given task, such as classification or regression.
Fidelity refers to the statistical similarity between real and its
corresponding synthetic data. This encompasses aspects like the distribution
of features across both datasets. Jordon et al. states that fidelity is often
considered alongside utility [25]. In our benchmarking evaluation, we
therefore considered Fidelity measures under Utility. In the evaluation
context, Privacy is a measure that quantifies the amount of information about
the original data revealed by its corresponding synthetic data. Table 2
provides a brief overview of the evaluation measures used.

Measure | Type of evaluation | Range | Optimal direction
---|---|---|---
Student’s t test | utility | $[0,1]$ | p-value ¿ 0.05
KL divergence | utility | $[0,+\infty]$ | smaller
Cross validation | utility | $[0,1]$ | smaller
Holdout data analysis | utility | $[0,1]$ | smaller
Propensity score | utility | $[0,0.25]$ | smaller
Log-cluster metric | utility | $[0,+\infty]$ | larger
Cross classification | utility | $[0,1]$ | smaller
Membership Inference Attack | privacy | $[0,1]$ | smaller
Attribute Inference Attack on categorical features | privacy | $[0,1]$ | smaller
Attribute Inference Attack on continuous features | privacy | $[0,+\infty]$ | smaller
Euclidean distance | privacy | $[0,+\infty]$ | larger
Hausdorff distance | privacy | $[0,+\infty]$ | larger
Cosine similarity | privacy | $[0,1]$ | smaller
Table 2:  Evaluation measures used for benchmarking analysis. This table
summarizes various evaluation measures to compare generative models. The first
column details the measure itself. the second column clarifies whether the
measure focuses on model utility or privacy. the third column specifies the
possible range of values for the measure. Finally, the last column indicates
the optimal direction for the score (smaller or larger) to signify better
performance.

###  4.3 Utility measures

We employed some common Utility measures to evaluate the quality of synthetic
data generated by generative models. In this section, we are going to provide
a brief description of the utility measures used for synthetic data
evaluation.

  * •

Student’s t-test:  In general, a t-test is a univariate statistical test used
for hypothesis testing to determine if there is a statistically significant
difference between the means of two groups. It is particularly useful for
comparing the means of two groups when the data is continuous and normally
distributed. Assuming the continuous features are normally distributed, the
student’s t-test is performed between the continuous attributes of real and
synthetic data, independent of other attributes [26]. This test is performed
for each continuous attribute with a significance level of $0.05$. The null
hypothesis ($h_{0}$) assumes that the distribution of real and synthetic
features has the same mean, while the alternate hypothesis ($h_{1}$) assumes
that the means of real and synthetic feature distributions are different. If
the p-value obtained from the test is higher than this value, the null
hypothesis ($h_{0}$) is accepted. Otherwise, the alternative hypothesis
($h_{1}$) is accepted. The properties analyzed in each test are preserved in
synthetic data if $h_{0}$ is accepted [26].

  * •

Kullback-Leibler (KL) divergence:  KL divergence is used to measure the
difference between two probability distributions. For our evaluation, we,
therefore, adapted this measure to quantify the similarity between the real
and synthetic data’s probability distributions of each categorical feature. A
zero value indicates identical distributions, while larger values signify a
greater discrepancy. It is important to note that, like the student’s t-test,
KL divergence is computed independently for each feature. This means that it
does not capture potential dependencies between features in the data [27, 26].

  * •

Machine learning efficacy:  To calculate this performance measure, first, two
Gradient-boosting classifiers are trained on real training data and its
corresponding synthetic data separately. Gradient boosting classifier is
employed due to its ability to combine weak models into a powerful learner
that effectively handles numerical and categorical features. After training,
both the models were tested on the same holdout data, which was not seen by
either the classifiers or the generative models during training. This is known
as holdout data analysis. The absolute difference in the performance (in terms
of the F1-score) was calculated. It is important to note that a five-fold
cross-validation was performed before training, and the absolute performance
difference between the models is calculated. The smaller the differences, the
better the machine learning efficiency of synthetic data. [28, 24].

  * •

Propensity score: The feasibility of training a machine learning
classification model to differentiate between real and synthetic data is
measured by propensity score [29, 28, 30]. Also known as the
distinguishability metric, the propensity score is a well-known utility
measure [31, 32, 33]. To calculate the propensity score, each sample is
assigned a binary indicator/label, with $1$ for real data and $0$ for
synthetic data. A logistic regression classifier is trained to predict the
likelihood of a given sample being real. The closer the predicted probability
is to $1$, the more likely the sample is a real record, while values closer to
$0$ indicate a synthetic record. The following equation defines the propensity
score:

$propensity=\frac{1}{n}\sum_{i=1}^{n}\left(p_{i}-0.5\right)^{2}$ (4)

where $n$ is the total number of sample and $p_{i}$ is the probability of the
$i$-th sample being real by the classifier.

A probability of $0.5$ means that the classifier cannot differentiate between
the real and synthetic data. Therefore, the probability mean square difference
would range between $0$ and $0.25$ if the two datasets were identical and
exceed $0.25$ if they were different. [26].

  * •

Log-cluster metric: The Log-cluster metric assesses whether a clustering
algorithm can differentiate between real and synthetic data in an unsupervised
manner. At first, real and synthetic data were labeled and concatenated. The
$k$-Means clustering algorithm is then applied to this concatenated data to
extract clusters. The following formula calculates the log cluster metric
score $M(X_{R},X_{S})$, indicating how well the clustering algorithm can
distinguish between real and synthetic data [27].

$M(X_{R},X_{S})=\log\left(\frac{1}{k}\sum_{i=1}^{k}\left[\frac{n^{R}_{i}}{n_{i}}-c\right]^{2}\right)$
(5)

Where: & k is the number of clusters, in our case, we extract two clusters,
n^R_i is the number of points in the i-th cluster from the real data,
n_i is the total number of points in the i-th cluster,
c is the proportion of real tuples in the merged dataset,
calculated as c = nRnR\+ nS, where n^S is the number of points from the
synthetic data.

A larger score in terms of magnitude indicates that a clustering algorithm
cannot differentiate between real and synthetic data.

  * •

Cross classification: The cross-classification is a metric that measures how
well the synthetic data captures the statistical dependency structure of real
data [30]. This measures the dependence via prediction accuracy (F1-score) of
one feature based on the features in the data. To calculate this metric, two
classifiers are trained on synthetic and real data separately, with each
categorical feature considered as a target/label [30]. The performance of both
models is then evaluated on holdout data, and the absolute difference between
their performance is calculated. The average absolute difference over the
features is reported in Section 5. If this difference is close to zero, it is
expected that the synthetic data is similar to real data in preserving the
statistical dependencies between the features.

###  4.4 Privacy measures

In benchmarking studies, it is often taken into account how well a synthetic
data generation model can preserve privacy. However, only a few privacy
measures are available. Moreover, there is no well-established standard
measure yet to quantify how much synthetic data preserves privacy. In our
benchmarking experiments, we used three privacy measures: Distance metrics,
Membership Inference Attack (MIA), and Attribute Inference Attack (AIA) [26,
27, 34]. If the synthetic data generated by a generative model is highly
similar to the real data, then the average distance between the pairwise data
points is expected to be very small. The smaller the distance, the higher the
potential for privacy risks. MIA occurs when an attacker tries to determine if
a patient record was employed in training a generative model. In contrast, AIA
happens when an attacker has access to some attributes or features of the real
data and tries to find the values of an unknown patient attribute using
synthetic data. In the following, we provide further information on these
measures.

  * •

Distance metrics: Hernadez et al. suggested three similarity measures to
evaluate how private synthetic data is compared to real data. The metrics
employed include Euclidean distance, Hausdorff distance, and cosine
similarity. The pairwise similarity value is calculated for each pair of
records between real and synthetic, and the mean values of those pairwise
similarity values are analyzed. Euclidean distance captures the difference
between two data points by calculating the straight-line distance in a
multidimensional space. A smaller Euclidean distance indicates a high risk of
privacy breach. Hausdorff distance calculates the maximum pairwise shortest
distance between real and synthetic data. A smaller Hausdorff distance is
associated with a higher privacy risk. Conversely, cosine similarity measures
the directional similarity between data points, reflecting how closely their
angles align in the multidimensional space. The higher the value of cosine
similarity, the higher the privacy risk.

  * •

MIA: This measure evaluates the re-identification risk in synthetic patient
data through a simulated attack scenario. The attacker has access to the
complete synthetic data and a random subset of real patient data (special case
where only 50% used for generative model training) ranging in availability
from 20% to 100% (It might contain samples that have been used for model
training). Following research by Goncalves et al., we compare real data
samples to the synthetic data samples using Hamming distance, a metric
indicating attribute differences [27]. If the distance falls below a
predefined threshold ($0.4$, $0.3$, $0.2$, or $0.1$), the attacker suspects a
match, suggesting a patient record from the real data was used to generate the
corresponding synthetic records. Precision, the proportion of correctly
identified matches, is calculated to assess the effectiveness of the attack.
According to Hernadez et al. and Mendelevitch et al., lower precision (below
$0.5$) indicates good privacy preservation, while higher values suggest an
increased risk of re-identification [34]. By plotting precision against the
percentage of accessible real data and the chosen threshold, researchers can
analyze the impact of attacker ability and similarity thresholds on the re-
identification risk of the synthetic data, which we have used later in Figure
3.

  * •

AIA: This measure investigates how attackers can use synthetic data to predict
missing sensitive attributes in real data. The attackers will have access to
complete synthetic data and quasi-identifiers for some real records. Quasi-
identifiers are attributes or features that are not unique identifiers, such
as age or gender. A decision tree classifier or regressor is trained on quasi-
identifiers of synthetic data to learn the sensitive feature or attribute
depending upon the target type [26]. These trained models are used to predict
the sensitive information of the real data with the quasi-identifier. The
model is evaluated by calculating accuracy for categorical features and root-
mean-squared error (RMSE) for numerical features. High accuracy or low RMSE
indicates the attacker successfully predicted missing attributes, posing a
high disclosure risk. Conversely, lower accuracy and higher RMSE suggest a low
risk of attackers inferring sensitive information using this approach [26,
34].

##  5 Results

This section uses tables and figures to present the evaluation results of
synthetic data generated by tabular data generative models across ten publicly
available clinical datasets. In the tables, if there is a cell filled with
NaN, it indicates that we could not sample synthetic data generated for that
dataset by that model to match the cardinality of the target column in the
real data. Therefore, it is not included in the evaluation. Furthermore, we
compared the performance as an average across the ten datasets for privacy
assessment. CTAB-GAN+ was not included in the privacy comparison because, for
certain datasets, the model could not match the cardinality of the synthetic
target column to the real data.

Table 3: Table displaying p-values for Student’s t-test comparing real and synthetic data’s continuous features across ten benchmarking datasets. $p>0.05$ (marked in bold fonts in the table) suggests agreement with the null hypothesis, indicating similarity in feature means between real and synthetic data. dataset | TVAE | CTGAN | CTAB-GAN | CTAB-GAN+ | TabDDPM | NextConvGeN
---|---|---|---|---|---|---
Heart failure | 0.1402 | 0.0894 | 0.0390 | 0.1563 | 0.0958 | 0.0349
Heart disease | 0.1274 | 0.0012 | 0.0499 | 0.0979 | 0.4482 | 0.0190
Lung cancer | NaN | 0.0441 | 0.0029 | 0.1755 | 0.5898 | 0.4915
Migraine | NaN | 0.1598 | 0.1393 | NaN | 0.4925 | 0.7074
Liver cirrhosis | NaN | 0.0657 | 0.0713 | 0.0918 | 0.1186 | 0.0375
Indian liver patients | 0.0728 | 0.0546 | 0.0377 | 0.1331 | 0.4163 | 0.7143
Pima Indian diabetes | 0.1994 | 0.0905 | 0.1263 | 0.1387 | 0.0280 | 0.6303
Contraceptive methods | 0.5098 | 0.0000 | 0.1584 | 0.1112 | 0.5991 | 0.0014
Obesity | 0.1630 | 0.0076 | 0.0001 | NaN | 0.6091 | 0.6555
Stroke | NaN | 0.0493 | 0.0000 | 0.0844 | 0.5409 | 0.3068
Table 4: The table illustrates the KL divergence measure between real and synthetic data’s categorical features across benchmark datasets. A lower value of the KL divergence score indicates a smaller difference between the real and synthetic data distributions. TabDDPM emerges as the top performer, consistently outperforming the other three models across most datasets. dataset | TVAE | CTGAN | CTAB-GAN | CTAB-GAN+ | TabDDPM | NextConvGeN
---|---|---|---|---|---|---
Heart failure | 0.1726 | 0.0056 | 0.0307 | 0.0077 | 0.0228 | 0.0059
Heart disease | 0.0617 | 0.0106 | 0.2300 | 0.0148 | 0.0051 | 0.0133
Lung cancer | NaN | 0.0134 | 0.0099 | 0.0081 | 0.0023 | 0.0018
Migraine | NaN | 0.0128 | 0.3568 | NaN | 0.0024 | 0.0030
Liver cirrhosis | NaN | 0.0098 | 0.2264 | 0.0082 | 0.0809 | 0.0045
Indian liver patients | 0.3083 | 0.0272 | 0.0606 | 0.0082 | 0.0004 | 0.0004
Pima Indian diabetes | 0.0000 | 0.0342 | 0.1383 | 0.0179 | 0.0117 | 0.0057
Contraceptive Methods | 0.1026 | 0.0092 | 0.2400 | 0.0088 | 0.0014 | 0.0029
Obesity | 0.0590 | 0.0314 | 0.3701 | NaN | 0.0010 | 0.0023
Stroke | NaN | 0.0215 | 0.1561 | 0.0034 | 0.0005 | 0.0029

TabDDPM and NextConvGeN preserve distributions of both continuous and
categorical features in synthetic data: We compared continuous feature
distributions in real and synthetic data by comparing average p-values across
the features by Student’s t-test (refer to Table 3). A high p-value (greater
than the significance level of $0.05$) indicates similarity between the
distributions. As shown in Table 3, the results reveal that for all datasets,
TVAE and CTAB-GAN+ yield p-values greater than $0.05$, suggesting that the
synthetic continuous feature distributions closely resemble the real ones.
However, due to mode collapse, we could not sample enough of some datasets to
maintain the cardinality of the target variable. The TabDDPM model produces a
p-value less than $0.05$ for only one dataset. Overall, TVAE, CTAB-GAN+, and
TabDDPM demonstrate similarities between the synthetic and original continuous
features. In the case of NextConvGeN, CTGAN, and CTAB-GAN, we find p-values
above the $0.05$ threshold for six, five, and four datasets, respectively.
Notably, NextConvGeN yields higher p-values for most datasets. Consequently,
we can infer that, after TVAE, CTAB-GAN+, and TabDDPM, the NextConvGeN model
generates synthetic samples with indistinguishable sample means from the
original samples.

Categorical features probability distributions across the synthetic and real
data were compared using KL divergence, and the findings are shown in Table 4.
A KL divergence score of zero signifies that the distributions are identical.
In contrast, higher scores indicate that the distributions are different.
TabDDPM proves to be the best-performing model, followed by NextConvGeN,
generating synthetic data with categorical features probability distributions
that are closer to the real data. However, one can observe that there are only
small differences in the performances of the TabDDPM and NextConvGeN models,
and they are quite comparable in terms of performance. TVAE, CTGAN, and CTAB-
GAN models achieved higher KL divergence scores consistently compared to other
models, indicating that the categorical probability distributions are not well
preserved. On the other hand, CTAB-GAN+ produces a smaller KL divergence score
compared to other GAN-based models and TVAE. Overall, the findings show that
TabDDPM and NextConvGeN are suitable for generating synthetic categorical
variables with similar probability distributions in real data.

Table 5: Table illustrating the log-cluster metric across ten benchmarked datasets. A higher magnitude value indicates increased challenges in effectively clustering real from synthetic data. TabDDPM outperforms other models with a larger average magnitude across datasets. dataset | TVAE | CTGAN | CTAB-GAN | CTAB-GAN+ | TabDDPM | NextConvGeN
---|---|---|---|---|---|---
Heart failure | -9.5402 | -7.8421 | -6.2545 | -11.3612 | -4.0491 | -2.0681
Heart disease | -8.7799 | -5.0021 | -2.3625 | -7.2651 | -10.9247 | -2.0670
Lung cancer | NaN | -6.1234 | -6.9445 | -8.3901 | -9.0033 | -9.3914
Migraine | NaN | -5.2308 | -1.6343 | NaN | -21.4164 | -21.4164
Liver Cirrhosis | NaN | -3.0929 | -2.0673 | -6.1925 | -2.7663 | -2.0703
Indian liver patients | -2.4319 | -4.9052 | -2.1909 | -5.8815 | -6.5057 | -8.9846
Pima Indian diabetes | -6.1012 | -8.9490 | -6.3372 | -6.9917 | -2.0017 | -12.7566
Contraceptive methods | -5.0534 | -6.5806 | -2.3289 | -7.1532 | -10.5784 | -9.2276
Obesity | -6.6065 | -7.7256 | -1.5535 | NaN | -11.0229 | -10.9935
Stroke | NaN | -9.5835 | -2.6115 | -7.7486 | -14.9799 | -9.1026
Average | -6.4189 | -6.5035 | -3.4285 | -7.6230 | -9.3248 | -8.8078

Synthetic data from TabDDPM preserves the distribution of clusters in real
data: Table 5 shows the log cluster values for various generative models
across ten publicly available data sets. This metric measures how similar the
distribution of clusters is across real and synthetic data. A higher magnitude
for this measure would indicate a similar distribution of clusters in real and
synthetic. As per Table 5, TabDDPM produces higher log cluster measures for
five of the ten benchmarking datasets, which is superior to any other model.
This indicates that the synthetic data generated by TabDDPM follows a similar
cluster distribution as that of real data. However, the performance of
NextConvGeN is comparable to TabDDPM, with the second highest average
magnitude of the log-cluster metric. This shows that the distribution of
clusters can be better retained in synthetic data by the models TabDDPM and
NextConvGeN.

Table 6: The table presents an absolute five-fold cross-validation performance difference between gradient-boosting classifiers trained on real and their corresponding synthetic data across benchmarking datasets. The lower the difference, the better the utility. NextConvGeN outperforms other models in most of the datasets.  dataset | TVAE | CTGAN | CTAB-GAN | CTAB-GAN+ | TabDDPM | NextConvGeN
---|---|---|---|---|---|---
Heart failure | 0.0154 | 0.1694 | 0.1770 | 0.1245 | 0.0755 | 0.0506
Heart disease | 0.0921 | 0.3373 | 0.3212 | 0.0483 | 0.0646 | 0.0367
Lung cancer | NaN | 0.0685 | 0.0601 | 0.0741 | 0.0258 | 0.0250
Migraine | NaN | 0.2928 | 0.2791 | NaN | 0.0597 | 0.0979
Liver cirrhosis | NaN | 0.1554 | 0.1258 | 0.0676 | 0.2071 | 0.1180
Indian liver patients | 0.0702 | 0.0227 | 0.0202 | 0.0434 | 0.1305 | 0.1071
Pima Indian diabetes | 0.1118 | 0.1448 | 0.1418 | 0.1169 | 0.0555 | 0.0969
Contraceptive methods | 0.1655 | 0.1523 | 0.1374 | 0.1575 | 0.0517 | 0.0457
Obesity | 0.2181 | 0.6895 | 0.6936 | NaN | 0.0278 | 0.0348
Stroke | NaN | 0.0008 | 0.0013 | 0.0018 | 0.0026 | 0.0044

Classifiers trained on real data and NextConvGeN, TabDDPM synthetic data shows
similar performance: Table 6 compares the absolute cross-validation difference
between the classifiers trained on real and synthetic data. The expected
cross-validation difference is smaller when the synthetic data is similar to
real data. We observe that out of ten benchmarking datasets, the NextConvGeN
and TabDDPM model outperforms other synthetic data generation models for three
datasets. The results indicate that even powerful classifiers like gradient
boosting perform similarly when trained on synthetic data (generated by
NextConvGeN or TabDDPM) and original data separately. On the other hand, the
results of TVAE, CTGAN, CTAB-GAN, and CTAB-GAN+ show more considerable
differences for most small clinical datasets compared to NextConvGeN and
TabDDPM, indicating that these models are more suited for large datasets.

Table 7: Table presenting the absolute difference in F1-score between classifiers trained on real and synthetic data with both models tested on the same holdout dataset across benchmarked datasets. This shows that the model trained on NextConvGeN synthetic data produces more consistent performance on unseen data.  dataset | TVAE | CTGAN | CTAB-GAN | CTAB-GAN+ | TabDDPM | NextConvGeN
---|---|---|---|---|---|---
Heart failure | 0.2337 | 0.3902 | 0.2252 | 0.2867 | 0.3181 | 0.0621
Heart disease | 0.0398 | 0.0568 | 0.1554 | 0.1265 | 0.1824 | 0.2338
Lung cancer | NaN | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000
Migraine | NaN | 0.2554 | 0.0374 | NaN | 0.0585 | 0.1365
Liver cirrhosis | NaN | 0.1044 | 0.1132 | 0.0687 | 0.1107 | 0.0904
Indian liver patients | 0.0000 | 0.2885 | 0.0968 | 0.0640 | 0.0000 | 0.0114
Pima Indian diabetes | 0.0736 | 0.1250 | 0.1454 | 0.1384 | 0.0930 | 0.0194
Contraceptive methods | 0.1135 | 0.1427 | 0.1192 | 0.1339 | 0.0754 | 0.0991
Obesity | 0.0958 | 0.0246 | 0.0379 | NaN | 0.0292 | 0.0432
Stroke | NaN | 0.2155 | 0.1432 | 0.2118 | 0.1794 | 0.2598

Assessment of predictive consistency of classifiers trained on synthetic data
on unseen data: To assess the comparability of classifiers trained on real and
synthetic data, we evaluate their ability to perform on unseen real data
(holdout data). To accomplish this, we compute the absolute difference in
performance on holdout data between the classifiers trained on real and
synthetic data, as detailed in Table 7. As the difference gets closer to zero,
it indicates that the classifiers trained on real and synthetic data are
similar, as reflected by an equivalent F1 score when tested on holdout data.
The results in the table demonstrate that NextConvGeN, TabDDPM, and CTAB-GAN
are comparable by slight differences in the three datasets. However, the TVAE,
CTGAN, and CTAB-GAN+ model performs better in two of ten benchmark datasets.
It is noteworthy that all models except TVAE show zero absolute difference in
the case of lung cancer data. We cannot conclusively say from the results
whether a particular synthetic data generation model has an edge over the
others.

Table 8: The table contains propensity scores for various generative models across benchmarking datasets. The closer the propensity score to 0, the higher the utility, indicating that it is difficult for machine learning classifiers to differentiate between real and synthetic samples. The results show that NextConvGeN can consistently generate synthetic data which is similar to real data compared. dataset | TVAE | CTGAN | CTAB-GAN | CTAB-GAN+ | TabDDPM | NextConvGeN
---|---|---|---|---|---|---
Heart failure | 0.0416 | 0.0586 | 0.0716 | 0.0330 | 0.1041 | 0.0033
Heart disease | 0.0949 | 0.1000 | 0.1618 | 0.1010 | 0.0080 | 0.0321
Lung cancer | NaN | 0.0649 | 0.0535 | 0.0615 | 0.0057 | 0.0049
Migraine | NaN | 0.0451 | 0.2123 | NaN | 0.0045 | 0.0048
Liver cirrhosis | NaN | 0.1083 | 0.0723 | 0.0830 | 0.9022 | 0.0041
Indian liver patients | 0.0836 | 0.1087 | 0.1158 | 0.0385 | 0.0052 | 0.0017
Pima Indian diabetes | 0.0689 | 0.0754 | 0.1105 | 0.0497 | 0.1360 | 0.0014
Contraceptive Methods | 0.0471 | 0.0268 | 0.1342 | 0.0106 | 0.0012 | 0.0045
Obesity | 0.0383 | 0.0486 | 0.1958 | NaN | 0.0014 | 0.0013
Stroke | NaN | 0.0185 | 0.1166 | 0.0079 | 0.0007 | 0.0021
Table 9: The table shows the absolute cross-classification scores difference between real and synthetic data for various models across the benchmarking datasets. A smaller difference signifies a stronger correlation between the features preserved in synthetic data. The findings indicate that NextConvGeN excels in maintaining inter-feature relationships compared to other models. dataset | TVAE | CTGAN | CTAB-GAN | CTAB-GAN+ | TabDDPM | NextConvGeN
---|---|---|---|---|---|---
Heart failure | 0.0667 | 0.1918 | 0.1518 | 0.1239 | 0.1107 | 0.0558
Heart disease | 0.0730 | 0.2168 | 0.2632 | 0.1754 | 0.1420 | 0.0644
Lung cancer | NaN | 0.0947 | 0.0914 | 0.1073 | 0.0205 | 0.0209
Migraine | NaN | 0.4165 | 0.4475 | NaN | 0.0241 | 0.0388
Liver cirrhosis | NaN | 0.1121 | 0.1288 | 0.0778 | 0.0884 | 0.0455
Indian liver patients | 0.0453 | 0.1095 | 0.0753 | 0.0777 | 0.0462 | 0.0247
Pima Indian diabetes | 0.0373 | 0.1188 | 0.1335 | 0.1314 | 0.0562 | 0.0364
Contraceptive methods | 0.0238 | 0.0906 | 0.1118 | 0.0835 | 0.0207 | 0.0261
Obesity | 0.3465 | 0.7695 | 0.7863 | NaN | 0.0301 | 0.0380
Stroke | NaN | 0.0059 | 0.0207 | 0.0075 | 0.0059 | 0.0063

The ability of convex space learning to generate synthetic data with high
utility: In the context of synthetic data generation, the propensity score is
the most commonly used utility measure for evaluating synthetic data [31, 32,
33]. The value for this measure lies between $0$ and $0.25$. If the value is
towards $0$, then it indicates two datasets (real and synthetic) are
identical, and the value towards $0.25$ shows they are different [30]. Table 8
presents the results of propensity scores for various models across the
benchmarking dataset. From the table, we can observe that NextConvGeN has a
propensity score close to zero, suggesting that it is challenging for the
machine learning classifiers to differentiate between real and NextConvGeN
generated synthetic data. This demonstrates that the convex space learning
technique can generate synthetic samples for small tabular clinical data
similar to the original data.

The results in Table 9 present the absolute cross-classification difference
for several generative models across ten benchmarking datasets. The cross-
correlation metric evaluates the statistical dependencies produced for a
variable by considering other variables through a classifier. The findings
reveal that NextConvGeN and TabDDPM perform at par by outperforming five out
of ten benchmarking datasets. These two models outperform GAN-based models and
TVAE. From Table 9, we can notice that NextConvGeN consistently produces small
differences close to zero. This consistency could be attributed to
NextConvGeN’s unique approach of treating categories in categorical variables
as real values, whereas other generative models employ one-hot encoded
vectors.

Overall, for utility measures, we observe that NextConvGeN and TabDDPM models
produce comparable performances for tabular clinical datasets of small size.
The following section discusses the performance of various generative models
for privacy breaches. Since we are evaluating privacy on an average of over
ten publicly available clinical datasets, we do not include TVAE and CTAB-GAN+
for comparison. For these models, we could not maintain the cardinality of the
synthetic target column as that of the real for some datasets.

Figure 2: Top Left Plot (Euclidean Distance): Box plot illustrating the
average Euclidean distance between real and synthetic datasets. Lower values
indicate better privacy preservation. The similarity across models suggests
comparable privacy risk. Top Right Plot (Hausdorff Distance): Box plot
depicting the Hausdorff distance between real and synthetic datasets. Higher
values indicate better privacy preservation. CTGAN exhibits values in a higher
range, implying lower privacy risk compared to other models. Bottom Plot
(Cosine Similarity): Box plot displaying the cosine similarity between real
and synthetic datasets. Lower values indicate better privacy preservation.
CTAB-GAN consistently demonstrates lower cosine distances, implying lower
privacy risk in terms of distance.

GAN-based models exhibit low risk of privacy breach in distance measure
evaluation: We evaluated the privacy of synthetic data using similarity
measures such as Euclidean distance, Hausdorff distance, and Cosine
similarity. These distances are calculated between real and its corresponding
synthetic data generated by various generative models across ten benchmarking
datasets. The generative models underwent five training rounds on real
training data with different random seeds, and we averaged the resulting
distances for each run to produce fifty values across the ten datasets. These
results were then visually represented through box plots showcasing the
distributions of Euclidean, Hausdorff, and Cosine similarities (refer to
Figure 2). The Euclidean distance (larger the better) box plot indicated that
privacy risks were relatively uniform across generative models. However, the
Hausdorff distance (larger the better) box plot highlighted lower privacy
risks with CTGAN due to greater distances between real and synthetic samples,
whereas other models showed smaller Hausdorff distances, indicating potential
privacy concerns. On the other hand, CTAB-GAN exhibited minimal cosine
similarity (smaller the better), indicating superior privacy protection, while
NextConvGeN and TabDDPM had broader distributions with extreme values,
suggesting possibly high privacy risks. In conclusion, while there were no
significant differences in Euclidean distance performance among models, CTGAN
and CTAB-GAN showed lower privacy risks regarding Hausdorff distance and
cosine similarity, respectively.

(a) CTGAN

(b) CTAB-GAN

(c) TabDDPM

(d) NextConvGeN

Figure 3: The line plots above depict precision values for membership
inference attacks against generative models. Each plot represents precision on
the y-axis and the proportion of data available to the attacker on the x-axis,
with different colored lines representing various thresholds. CTGAN, CTAB-GAN,
and TabDDPM consistently exhibit precision values below $0.5$ across different
access proportions and thresholds, indicating a low risk of reidentification.
Conversely, NextConvGeN’s plot shows precision values around $0.6$ across
varying access levels and thresholds, suggesting an increased risk of
reidentification.

GAN-based models are less sensitive to MIA and AIA: Figure 3 illustrates the
average precision score for MIA for the generative models in the benchmarking
study. The lower the precision score, the lower the risk of privacy breach.
While CTGAN, CTAB-GAN, and TabDDPM consistently exhibit precision scores below
$0.5$ across different thresholds and access levels, NextConvGeN achieves a
precision score of around $0.6$. Thus, NextConvGeN performs poorly in terms of
MIA.

Figure 4: The box plots above showcase the results of the Attribute Inference
Attack across different models. For categorical features, F1-scores were
calculated, where lower values indicate a lower reidentification risk (left
box plot). Root Mean Square (RMS) values were computed for continuous
features, with larger values suggesting a lower reidentification risk (right
box plot). The plots reveal that CTGAN and CTAB-GAN consistently exhibit lower
F1-scores and higher RMS values distributions, indicating a lower
reidentification risk compared to TabDDPM and NextConvGeN.

The results of the Attribute Inference Attack were analyzed using box plots to
plot the F1-score for predicting sensitive categorical features and the RMS
value for predicting sensitive continuous features, as shown in Figure 4. A
lower F1 score indicates a greater difficulty in predicting the unknown
variable. In contrast, a higher RMS value indicates a greater challenge for
the machine learning model in estimating the missing sensitive variable. The
analysis revealed that the CTGAN and CTAB-GAN models did not perform well in
predicting sensitive features when trained on quasi-identifiers, resulting in
a higher range of RMS value for continuous sensitive features and a
comparatively lower range of F1-score for categorical features, indicating a
low-risk of individual patient reidentification. Conversely, the NextConvGeN
and TabDDPM models exhibited a high risk of attribute inference.

To sum up, we observe that the NextConvGeN and the TabDDPM models generate
more realistic synthetic data. The synthetic data generated from such models
can better mimic the real data regarding utility-based tasks. For example, if
synthetic data from NextConvGeN is used for a classification task, then the
classification performance would be relatively similar to the situation if the
real data were used for the same classification task. For privacy
preservation, however, we notice that CTGAN and CTAB-GAN perform superiorly.

##  6 Discussion

We will now discuss some interesting observations that arise from our results.
Our result reflects the dilemma of designing a synthetic data generation
strategy that preserves privacy and utility. High value for utility measures
broadly indicate that the statistical distributions of the real and the
synthetic data are similar. Moreover, it also indicates that the performance
of machine learning tasks such as clustering or classification is similar
across real and synthetic data. If the synthetic data is to be used as an
alternative to real data, then high utility is required.

In Table 6, notice the absolute five-fold cross-validation performance
difference between the gradient descent classifier trained on real and
synthetic data. For the CTGAN and CTAB-GAN models, the difference is more than
ten percent for more than fifty percent of the datasets. There is no objective
measure in the literature that comments on the effect of such difference once
these synthetic data generation strategies are applied to practical, real-life
scenarios. More than a ten percent difference on more than half the datasets
intuitively translates to poorly performing models. However, if we focus too
much on making the synthetic data similar enough to the real data, we
automatically compromise on privacy. Once the data distribution of the
synthetic data is too similar to that of the real data, it is easier to track
the real data back from the synthetic data (reidentification risk). This is
reflected in our results. We observe that TabDDPM and NextConvGeN models
perform worse than CTGAN and CTAB-GAN regarding privacy measures. Therefore,
our results reflect the utility-privacy dilemma.

Note that GAN-based models fundamentally rely on adversarial learning. Such
models fundamentally try to learn a mapping from a random distribution (noise)
to the data distribution. The real data is no longer needed for the synthetic
sample generation step in such models. A random noise picked from the chosen
noise distribution can be converted to a real data point. On the other hand,
both TabDDPM and NextConvGeN use the real data points directly for the
synthetic data generation. In NextConvGeN, for example, a data neighborhood
sampled from a real data neighborhood is passed on to the generator to
generate synthetic samples from the convex hull of that data neighborhood.
This can be a key driving factor behind the high-utility performance of the
TabDDPM and the NextConvGeN models. The poor utility performance of CTGAN,
CTAB-GAN, and CTAB-GAN+ can be even more pronounced when the training dataset
is smaller. We can expect the GAN models to require more training data, as
they are essentially learning to map a completely random distribution to a
known data distribution. In real-life scenarios, however, clinical data,
particularly generated at a single health facility, is often small in size.

A natural question we can ask here is that which among the privacy and utility
measures are more important. Unfortunately, not much research has been
conducted to answer such a question. However, common sense suggests that a
synthetic data generation model that performs poorly in terms of utility in
the first place is of little value. Even though it preserves privacy, the
synthetic data does not represent the real data well enough. Therefore, it is
unsuitable for any machine learning analysis that is performed to make
decisions based on real data. On the other hand, if at least the utility is
high, then there are measures for the safe sharing of data in general, which
can be used for sharing the synthetic data [35]. In such a case, the high-
utility synthetic data can provide an additional level of data security. We
would argue that a synthetic data generation model can be judged in terms of
privacy, given that it satisfies utility requirements to some degree. We point
out the necessity of an objective determination of a proper utility threshold
for a synthetic data generation model to be considered for privacy
preservation.

In this study, we would like to highlight the mode collapse problem in TVAE
and CTAB-GAN+. Mode collapse occurs when generative models converge on a
limited set of outputs, failing to capture the full diversity of the data.
Although CTAB-GAN+ is designed with a specific architectural feature to
address mode collapse (training by sampling and packing), its performance on
small clinical data has not been evaluated. The mode collapse in CTAB-GAN+
might be attributed to the limited training data. With a smaller dataset, the
generators may struggle to learn the complex underlying distributions of
minority samples, resulting in a preference for majority samples. In TVAE,
information compression in the latent space during encoding could hinder the
decoder’s ability to reconstruct all target classes. On the other hand, in
NextConvGeN, the generators are trained in every minority sample neighborhood
and generate samples within each neighborhood. Overall, the evaluation study
suggests that TVAE and CTAB-GAN+ may not be optimal if the dataset is small
and imbalanced.

There are many generative models that lack some practical aspects, such as the
preservation of functional and logical dependencies in synthetic data.
Typically, the features in tabular datasets can be logically or functionally
dependent. Let us take a crude example to understand this. Say a tabular
dataset has two features: BMI (real-valued) and Diabetes (binary-valued). Say
the dataset is curated in a health facility that considers a BMI threshold of
$t$ for the diagnosis of Diabetes. That is, if patients have a BMI of more
than $t$, they are more likely to be diabetic. There is, therefore, a logical
dependency between the two features of BMI and Diabetes. In synthetic data, if
there is a data point for this, the BMI is greater than the threshold, but the
Diabetes value is negative, then a logical fallacy arises. Notably, such
intricate complicates arise in synthetic data generation for the tabular data
modality, demanding more research on this topic.

We summarize a few observations that we believe require attention by the
community:

  * •

The objective assessment of generative model performance remains a key
challenge in the field. For example, what degree of performance is good enough
for a synthetic data generation model to be labelled as a high-utility model
is not clear. Such clarity can be attained when more such models are used in
real-life scenarios. An objective measure to quantify the utility-privacy
balance must also be devised to compare the versatility of the synthetic data
generation models.

  * •

From our study, we noticed that different synthetic data generation strategies
are apt for different tasks. The purpose of synthetic data generation can be
diverse. For example, synthetic data can be used to mimic a classification or
clustering of real data. Other purposes can be circumventing privacy issues or
maintaining logical dependencies. A single synthetic data generation model is
unlikely to serve all these purposes. In such a case, the idea of task-
specific synthetic data generation can be explored. The idea is to develop
synthetic data generation models specialized for different tasks.

  * •

Several synthetic data generation philosophies, such as adversarial learning,
diffusion models, and convex space learning, have been used to develop the
existing synthetic data generation models. We noticed, for example, that the
diffusion model and the convex space learning philosophy work well for utility
purposes, whereas the adversarial philosophy works better for privacy
preservation, although it significantly compromises utility. A hybrid approach
considering different synthetic data generation philosophies can also be
speculated to produce a more balanced utility-privacy performance.

##  7 Conclusion

We proposed the NextConvGeN model, a deep learning architecture for generating
entire tabular data. The model improves on previous convex space learning
(ConvGeN) model built for oversampling on imbalanced datasets. However, there
are some important additions to our new model’s synthetic data generation
process. Firstly, we used an improved approach to identify data neighborhoods.
This approach involves separate dimension reduction for each feature type
using appropriate distance measures, leading to a more informative lower-
dimensional representation of the real data. This allows NextConvGeN to
capture non-linear similarities when determining data neighborhoods. Secondly,
we added the ‘Alpha clipping,’ a technique to control the variance of the
synthetic samples generated in a data neighborhood. It also prevents the
synthetic data generated from being too similar to the real data. Thirdly, we
adapt the training procedure of NextConvGeN outside the context of imbalanced
classification. In contrast to ConvGeN, which receives a minority neighborhood
as input to the generator, which is discriminated against a batch of majority
class samples, NextConvGeN receives an FDC-assisted data neighborhood, which
is discriminated against a random batch drawn from the rest of the data
distribution (complement of the FDC-assisted data neighborhood).

In general, the performances of NextConvGeN are comparable to the TabDDPM
model. However, NextConvGeN shows superior performance for utility measures.
For classification tasks, we saw that synthetic samples generated by
NextConvGeN preserve classification performance for synthetic data compared to
real data even better compared to TabDDPM. After comparing various synthetic
data generation approaches, we delve into the critical challenge of balancing
utility and privacy, an aspect overlooked in previous research. Finally, in
Section 6, we address the limitations of our approach and explore the broader
context of this field. Our work, therefore, can be a valuable addition to the
literature on synthetic tabular data generation. The proposed model has the
potential for wider applications in utility-focused synthetic data generation
and balanced privacy.

## Conflict of Interest

The authors have no conflict of interest.

## Availability of code and results

To support transparency, re-usability, and reproducibility, we provided
detailed Jupyter notebooks from our experiments in
https://github.com/manjunath-mahendra/NextConvGeNGitHub.

## Acknowledgment

This work has been supported by the German Research Foundation (DFG), FK
515800538 obtained for ‘Learning convex data spaces for generating synthetic
clinical tabular data’.

## References

  * [1] S. Bej, K. Schulz, P. Srivastava, M. Wolfien, O. Wolkenhauer, A multi-schematic classifier-independent oversampling approach for imbalanced datasets, IEEE Access 9 (2021) 123358–123374.  https://doi.org/10.1109/ACCESS.2021.3108450 doi:10.1109/ACCESS.2021.3108450.
  * [2] S. Bej, N. Davtyan, M. Wolfien, M. Nassar, O. Wolkenhauer, LoRAS: an oversampling approach for imbalanced datasets, Machine Learning 110 (2) (2021) 279–301.  https://doi.org/10.1007/s10994-020-05913-4 doi:10.1007/s10994-020-05913-4.
  * [3] K. Schultz, S. Bej, W. Hahn, M. Wolfien, P. Srivastava, O. Wolkenhauer, ConvGeN: A convex space learning approach for deep-generative oversampling and imbalanced classification of small tabular datasets, Pattern Recognition 147 (2024) 110138.  https://doi.org/10.1016/j.patcog.2023.110138 doi:10.1016/j.patcog.2023.110138.
  * [4] D. P. Kingma, M. Welling, An introduction to variational autoencoders, Foundations and Trends® in Machine Learning 12 (4) (2019) 307–392.  https://doi.org/10.1561/2200000056 doi:10.1561/2200000056.
  * [5] M. Pesteie, P. Abolmaesumi, R. N. Rohling, Adaptive augmentation of medical data using independently conditional variational auto-encoders, IEEE Transactions on Medical Imaging 38 (12) (2019) 2807–2820.  https://doi.org/10.1109/TMI.2019.2914656 doi:10.1109/TMI.2019.2914656.
  * [6] A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, A. A. Bharath, Generative adversarial networks: An overview, IEEE Signal Processing Magazine 35 (1) (2018) 53–65.  https://doi.org/10.1109/MSP.2017.2765202 doi:10.1109/MSP.2017.2765202.
  * [7] A. Figueira, B. Vaz, Survey on synthetic data generation, evaluation methods and GANs, Mathematics 10 (15) (2022) 2733.  https://doi.org/10.3390/math10152733 doi:10.3390/math10152733.
  * [8] F.-A. Croitoru, V. Hondru, R. T. Ionescu, M. Shah, Diffusion models in vision: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (9) (2023) 10850–10869.  https://doi.org/10.1109/TPAMI.2023.3261988 doi:10.1109/TPAMI.2023.3261988.
  * [9] V. Borisov, T. Leemann, K. Seßler, J. Haug, M. Pawelczyk, G. Kasneci, Deep neural networks and tabular data: A survey, IEEE Transactions on Neural Networks and Learning Systems (2022) 1–21https://doi.org/10.1109/TNNLS.2022.3229161 doi:10.1109/TNNLS.2022.3229161.
  * [10] H. Han, W.-Y. Wang, B.-H. Mao, Borderline-SMOTE: A new over-sampling method in imbalanced data sets learning, in: D.-S. Huang, X.-P. Zhang, G.-B. Huang (Eds.), Advances in Intelligent Computing, Lecture Notes in Computer Science, Springer, 2005, pp. 878–887.  https://doi.org/10.1007/11538059_91 doi:10.1007/11538059_91.
  * [11] S. Kamthe, S. Assefa, M. Deisenroth, Copula flows for synthetic data generation (2021).  http://arxiv.org/abs/2101.00598 [cs, stat] arXiv:2101.00598 [cs, stat], https://doi.org/10.48550/arXiv.2101.00598 doi:10.48550/arXiv.2101.00598.
  * [12] E. Choi, S. Biswal, B. Malin, J. Duke, W. F. Stewart, J. Sun, Generating multi-label discrete patient records using generative adversarial networks, in: Proceedings of the 2nd Machine Learning for Healthcare Conference, PMLR, 2017, pp. 286–305, ISSN: 2640-3498.
  * [13] S. Sirur, J. R. Nurse, H. Webb, Are we there yet? understanding the challenges faced in complying with the general data protection regulation (GDPR), in: Proceedings of the 2nd International Workshop on Multimedia Privacy and Security, MPS ’18, Association for Computing Machinery, 2018, pp. 88–95.  https://doi.org/10.1145/3267357.3267368 doi:10.1145/3267357.3267368.
  * [14] L. Xu, M. Skoularidou, A. Cuesta-Infante, K. Veeramachaneni, Modeling tabular data using conditional GAN, in: H. Wallach, H. Larochelle, A. Beygelzimer, F. d. Alché-Buc, E. Fox, R. Garnett (Eds.), Advances in Neural Information Processing Systems, Vol. 32, Curran Associates, Inc., 2019.
  * [15] Z. Zhao, A. Kunar, R. Birke, L. Y. Chen, CTAB-GAN: Effective table data synthesizing, in: Proceedings of The 13th Asian Conference on Machine Learning, PMLR, 2021, pp. 97–112, ISSN: 2640-3498.
  * [16] A. Kotelnikov, D. Baranchuk, I. Rubachev, A. Babenko, TabDDPM: Modelling tabular data with diffusion models, in: Proceedings of the 40th International Conference on Machine Learning, PMLR, 2023, pp. 17564–17579, ISSN: 2640-3498.
  * [17] M. Sami, I. Mobin, A comparative study on variational autoencoders and generative adversarial networks, in: 2019 International Conference of Artificial Intelligence and Information Technology (ICAIIT), 2019, pp. 1–5.  https://doi.org/10.1109/ICAIIT.2019.8834544 doi:10.1109/ICAIIT.2019.8834544.
  * [18] A. Vahdat, J. Kautz, NVAE: A deep hierarchical variational autoencoder, in: Advances in Neural Information Processing Systems, Vol. 33, Curran Associates, Inc., 2020, pp. 19667–19679.
  * [19] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial networks, Communications of the ACM 63 (11) (2020) 139–144.  https://doi.org/10.1145/3422622 doi:10.1145/3422622.
  * [20] K. Wang, C. Gou, Y. Duan, Y. Lin, X. Zheng, F.-Y. Wang, Generative adversarial networks: introduction and outlook, IEEE/CAA Journal of Automatica Sinica 4 (4) (2017) 588–598.  https://doi.org/10.1109/JAS.2017.7510583 doi:10.1109/JAS.2017.7510583.
  * [21] Z. Zhao, A. Kunar, R. Birke, H. Van der Scheer, L. Y. Chen, CTAB-GAN+: enhancing tabular data synthesis, Frontiers in Big Data 6 (2023) 1296508.  https://doi.org/10.3389/fdata.2023.1296508 doi:10.3389/fdata.2023.1296508.
  * [22] S. Bej, C. Umesh, M. Mahendra, K. Schultz, J. Sarkar, O. Wolkenhauer, Accounting for diverse feature-types improves patient stratification on tabular clinical datasets, Machine Learning with Applications 14 (2023) 100490\.  https://doi.org/10.1016/j.mlwa.2023.100490 doi:10.1016/j.mlwa.2023.100490.
  * [23] S. Bej, J. Sarkar, S. Biswas, P. Mitra, P. Chakrabarti, O. Wolkenhauer, Identification and epidemiological characterization of type-2 diabetes sub-population using an unsupervised machine learning approach, Nutrition & Diabetes 12 (1) (2022) 1–11.  https://doi.org/10.1038/s41387-022-00206-2 doi:10.1038/s41387-022-00206-2.
  * [24] M. Hernandez, G. Epelde, A. Alberdi, R. Cilla, D. Rankin, Synthetic data generation for tabular health records: A systematic review, Neurocomputing 493 (2022) 28–45.  https://doi.org/10.1016/j.neucom.2022.04.053 doi:10.1016/j.neucom.2022.04.053.
  * [25] J. Jordon, L. Szpruch, F. Houssiau, M. Bottarelli, G. Cherubin, C. Maple, S. N. Cohen, A. Weller, Synthetic data – what, why and how? (2022).  http://arxiv.org/abs/2205.03257 [cs] arXiv:2205.03257 [cs], https://doi.org/10.48550/arXiv.2205.03257 doi:10.48550/arXiv.2205.03257.
  * [26] M. Hernadez, G. Epelde, A. Alberdi, R. Cilla, D. Rankin, Synthetic tabular data evaluation in the health domain covering resemblance, utility, and privacy dimensions, Methods of Information in Medicine 62 (2023) e19–e38.  https://doi.org/10.1055/s-0042-1760247 doi:10.1055/s-0042-1760247.
  * [27] A. Goncalves, P. Ray, B. Soper, J. Stevens, L. Coyle, A. P. Sales, Generation and evaluation of synthetic patient data, BMC Medical Research Methodology 20 (1) (2020) 108.  https://doi.org/10.1186/s12874-020-00977-1 doi:10.1186/s12874-020-00977-1.
  * [28] E. Espinosa, A. Figueira, On the quality of synthetic generated tabular data, Mathematics 11 (15) (2023) 3278, multidisciplinary Digital Publishing Institute.  https://doi.org/10.3390/math11153278 doi:10.3390/math11153278.
  * [29] A. Pathare, R. Mangrulkar, K. Suvarna, A. Parekh, G. Thakur, A. Gawade, Comparison of tabular synthetic data generation techniques using propensity and cluster log metric, International Journal of Information Management Data Insights 3 (2) (2023) 100177.  https://doi.org/10.1016/j.jjimei.2023.100177 doi:10.1016/j.jjimei.2023.100177.
  * [30] F. K. Dankar, M. K. Ibrahim, L. Ismail, A multi-dimensional evaluation of synthetic data generators, IEEE Access 10 (2022) 11147–11158.  https://doi.org/10.1109/ACCESS.2022.3144765 doi:10.1109/ACCESS.2022.3144765.
  * [31] F. K. Dankar, M. Ibrahim, Fake it till you make it: Guidelines for effective synthetic data generation, Applied Sciences 11 (5) (2021) 2158, multidisciplinary Digital Publishing Institute.  https://doi.org/10.3390/app11052158 doi:10.3390/app11052158.
  * [32] J. Drechsler, Synthetic Datasets for Statistical Disclosure Control: Theory and Implementation, Springer Science & Business Media, 2011.
  * [33] G. M. Raab, B. Nowok, C. Dibben, Guidelines for producing useful synthetic data (2017).  https://doi.org/10.48550/arXiv.1712.04078 doi:10.48550/arXiv.1712.04078.
  * [34] O. Mendelevitch, M. D. Lesh, Fidelity and privacy of synthetic medical data (2021).  http://arxiv.org/abs/2101.08658 [cs] arXiv:2101.08658 [cs], https://doi.org/10.48550/arXiv.2101.08658 doi:10.48550/arXiv.2101.08658.
  * [35] I. Gupta, A. Singh, C.-N. Lee, R. Buyya, Secure data storage and sharing techniques for data protection in cloud environments: A systematic review, analysis, and future directions, IEEE Access 10 (2022-01-01) 1–1.  https://doi.org/10.1109/ACCESS.2022.3188110 doi:10.1109/ACCESS.2022.3188110.

thebibliography00

* [36] Kristian Schultz, Saptarshi Bej, Waldemar Hahn, Markus Wolfien, Prashant Srivastava, Olaf Wolkenhauer, ConvGeN: A convex space learning approach for deep-generative oversampling and imbalanced classification of small tabular datasets,_Pattern Recognition_ , 2024, volume 147, pages 110138, ISSN 0031-3203, https://doi.org/10.1016/j.patcog.2023.110138
* [37] Saptarshi Bej, Kristian Schulz, Prashant Srivastava, Markus Wolfien, and Olaf Wolkenhauer, A multi-schematic classifier-independent oversampling approach for imbalanced datasets.,_IEEE Access_ , 2021, volume 9, pages 123358-123374, ISSN 2169-3536, https://doi.org/10.1109/ACCESS.2021.3108450
* [38] Borisov, Vadim and Leemann, Tobias and Seßler, Kathrin and Haug, Johannes and Pawelczyk, Martin and Kasneci, Gjergji, Deep neural networks and tabular data: A survey, _IEEE Transactions on Neural Networks and Learning Systems_ , 2022: 1-21, https://doi.org/10.1109/TNNLS.2022.3229161
* [39] Han, Hui and Wang, Wen-Yuan and Mao, Bing-Huan, Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning, _Lecture Notes in Computer Science_ , Springer, Berlin, Heidelberg, 2005, 3644, https://doi.org/10.1007/11538059_91
* [40] Saptarshi Bej, Chaithra Umesh, Manjunath Mahendra, Kristian Schultz, Jit Sarkar, Olaf Wolkenhauer, Accounting for diverse feature-types improves patient stratification on tabular clinical datasets, _Machine Learning with Applications_ , Volume 14, 2023, 100490, ISSN 2666-8270, https://doi.org/10.1016/j.mlwa.2023.100490
* [41] Jane S. Saczynski, David D. McManus, Robert J. Goldberg, Commonly Used Data-collection Approaches in Clinical Research, _The American Journal of Medicine_ , Volume 126, Issue 11, 2013, Pages 946-950, ISSN 0002-9343, https://doi.org/10.1016/j.amjmed.2013.04.016
* [42] Sanket Kamthe and Samuel Assefa and Marc Deisenroth, arXiv, 2021, 2101.00598, https://doi.org/10.48550/arXiv.2101.00598
* [43] Choi, Edward and Biswal, Siddharth and Malin, Bradley and Duke, Jon and Stewart, Walter F. and Sun, Jimeng, Generating Multi-label Discrete Patient Records using Generative Adversarial Networks, _Proceedings of the 2nd Machine Learning for Healthcare Conference,_ , 2017, volume 68, pages 286-305, ISSN 0002-9343, https://proceedings.mlr.press/v68/choi17a.html
* [44] Sirur, Sean and Nurse, Jason R.C. and Webb, Helena, Are We There Yet? Understanding the Challenges Faced in Complying with the General Data Protection Regulation (GDPR), _Proceedings of the 2nd International Workshop on Multimedia Privacy and Security,_ 2018, pages 88–95, https://doi.org/10.1145/3267357.3267368
* [45] Kotelnikov, Akim and Baranchuk, Dmitry and Rubachev, Ivan and Babenko, Artem, TabDDPM: Modelling Tabular Data with Diffusion Models, _Proceedings of the 40th International Conference on Machine Learning_ , PMLR, 2023, volume 202, pages 17564-17579, https://proceedings.mlr.press/v202/kotelnikov23a.html
* [46] Diederik P. Kingma and Max Welling, An Introduction to Variational Autoencoders, _Foundations and Trends® in Machine Learning_ , 2019, volume 12, issue 4, pages 307-392, http://dx.doi.org/10.1561/2200000056
* [47] Figueira, Alvaro, and Bruno Vaz, Survey on Synthetic Data Generation, Evaluation Methods and GANs, _Mathematics_ , 2022, volume 10, issue 15, 2733, https://doi.org/10.3390/math10152733
* [48] Gui, Jie, Zhenan Sun, Yonggang Wen, Dacheng Tao, and Jieping Ye, A review on generative adversarial networks: Algorithms, theory, and applications, _IEEE transactions on knowledge and data engineering_ , 2021, volume 35, issue 4, pages 3313-3332, https://doi.org/10.1109/TKDE.2021.3130191
* [49] Croitoru, Florinel-Alin and Hondru, Vlad and Ionescu, Radu Tudor and Shah, Mubarak, Diffusion models in vision: A survey, _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023, volume 45, issue 9, pages 10850-10869, https://doi.org/10.1109/TPAMI.2023.3261988
* [50] Luo, Renqian, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu, BioGPT: generative pre-trained transformer for biomedical text generation and mining, _Briefings in Bioinformatics_ , 2022, volume 23, issue 6, pages 10850-10869, https://doi.org/10.1093/bib/bbac409
* [51] Pesteie, Mehran, Purang Abolmaesumi, and Robert N. Rohling, Adaptive augmentation of medical data using independently conditional variational auto-encoders, _IEEE transactions on medical imaging_ , 2022, volume 38, issue 12, pages 2807-2820, https://doi.org/10.1109/TMI.2019.2914656
* [52] Liu, Xiao, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang, GPT understands, too, _AI Open_ , 2023, https://doi.org/10.1016/j.aiopen.2023.08.012
* [53] Kamthe, Sanket, Samuel Assefa, and Marc Deisenroth, Copula flows for synthetic data generation, _arXiv preprint_ , 2021, https://doi.org/10.48550/arXiv.2101.00598
* [54] Park, Noseong, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Youngmin Kim, Data synthesis based on generative adversarial networks, _Proceedings of the VLDB Endowment_ , 2018, volume 11, issue 10, pages 1071-1083, https://doi.org/10.14778/3231751.3231757
* [55] Goodfellow, Ian and Pouget-Abadie, Jean and Mirza, Mehdi and Xu, Bing and Warde-Farley, David and Ozair, Sherjil and Courville, Aaron and Bengio, Yoshua, Generative adversarial networks, _Commun. ACM_ , Association for Computing Machinery, 2020, volume 63, issue 11, pages 139–144, https://doi.org/10.1145/3422622
* [56] Wang, Kunfeng and Gou, Chao and Duan, Yanjie and Lin, Yilun and Zheng, Xinhu and Wang, Fei-Yue, Generative adversarial networks: introduction and outlook, _IEEE/CAA Journal of Automatica Sinica_ , 2017, volume 4, issue 4, pages 588-598, https://doi.org/10.1109/JAS.2017.7510583
* [57] Creswell, Antonia and White, Tom and Dumoulin, Vincent and Arulkumaran, Kai and Sengupta, Biswa and Bharath, Anil A, Generative Adversarial Networks: An Overview, _IEEE Signal Processing Magazine_ , 2018, volume 35, issue 1, pages 53-65, https://doi.org/10.1109/MSP.2017.2765202
* [58] Xu, Lei, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni, Modeling tabular data using conditional gan, _Advances in neural information processing systems_ , 2019, volume 32
* [59] Zhao, Zilong and Kunar, Aditya and Birke, Robert and Chen, Lydia Y, CTAB-GAN: Effective Table Data Synthesizing, _Proceedings of The 13th Asian Conference on Machine Learning_ , Proceedings of Machine Learning Research, 2021, volume 157, pages 97-112, https://proceedings.mlr.press/v157/zhao21a.html
* [60] Bej, Saptarshi, Narek Davtyan, Markus Wolfien, Mariam Nassar, and Olaf Wolkenhauer, LoRAS: An oversampling approach for imbalanced datasets, _Machine Learning_ , 2021, volume 110, issue 2, pages 279-301, https://doi.org/10.1007/s10994-020-05913-4
* [61] S. Bej, K. Schulz, P. Srivastava, M. Wolfien and O. Wolkenhauer, A Multi-Schematic Classifier-Independent Oversampling Approach for Imbalanced Datasets,_IEEE Access_ , 2021, volume 9, pages 123358-123374, https://doi.org/10.1109/ACCESS.2021.3108450
* [62] Saptarshi Bej, Jit Sarkar, Saikat Biswas, Pabitra Mitra, Partha Chakrabarti, and Olaf Wolkenhauer, Identification and epidemiological characterization of Type-2 diabetes sub-population using an unsupervised machine learning approach, _Nutrition & Diabetes_, 2022, volume 12, issue 1, pages 1-11, https://doi.org/10.1038/s41387-022-00206-2
* [63] Sami, Mirza and Mobin, Iftekharul, A Comparative Study on Variational Autoencoders and Generative Adversarial Networks, _International Conference of Artificial Intelligence and Information Technology (ICAIIT)_ , 2019, pages 1-5, https://doi.org/10.1109/ICAIIT.2019.8834544
* [64] Vahdat, Arash, and Jan Kautz, NVAE: A deep hierarchical variational autoencoder, _Advances in neural information processing systems_ , 2020, volume 33, pages 19667-19679
* [65] Xu, Lei, and Kalyan Veeramachaneni, Synthesizing tabular data using generative adversarial networks, _arXiv preprint_ , 2018, https://doi.org/10.48550/arXiv.1811.11264
* [66] Armanious, Karim, Chenming Jiang, Marc Fischer, Thomas Küstner, Tobias Hepp, Konstantin Nikolaou, Sergios Gatidis, and Bin Yang, MedGAN: Medical image translation using GANs, _Computerized medical imaging and graphics_ , 2020, volume 79, pages 101684, https://doi.org/10.1016/j.compmedimag.2019.101684
* [67] La Salvia, Marco, Emanuele Torti, Raquel Leon, Himar Fabelo, Samuel Ortega, Beatriz Martinez-Vega, Gustavo M. Callico, and Francesco Leporati, Deep Convolutional Generative Adversarial Networks to Enhance Artificial Intelligence in Healthcare: A Skin Cancer Application, _Sensors_ , 2022, volume 22, issue 16, pages 6145, https://doi.org/10.3390/s22166145
* [68] Hernandez, Mikel, Gorka Epelde, Ane Alberdi, Rodrigo Cilla, and Debbie Rankin, Synthetic data generation for tabular health records: A systematic review, _Neurocomputing_ , 2022, volume 493, pages 28-45, https://doi.org/10.1016/j.neucom.2022.04.053
* [69] Fonseca, Joao, and Fernando Bacao, Tabular and latent space synthetic data generation: a literature review, _Journal of Big Data_ , 2023, volume 10, issue 1, pages 115, https://doi.org/10.1186/s40537-023-00792-7
* [70] Pathare, Aryan, Ramchandra Mangrulkar, Kartik Suvarna, Aryan Parekh, Govind Thakur, and Aruna Gawade, Comparison of tabular synthetic data generation techniques using propensity and cluster log metric, _International Journal of Information Management Data Insights_ , 2023, volume 3, issue 2, https://doi.org/10.1016/j.jjimei.2023.100177
* [71] Goncalves, Andre, Priyadip Ray, Braden Soper, Jennifer Stevens, Linda Coyle, and Ana Paula Sales, Generation and evaluation of synthetic patient data, _BMC medical research methodology_ , 2020, volume 20, issue 1, https://doi.org/10.1186/s12874-020-00977-1
* [72] Dankar, Fida K. and Ibrahim, Mahmoud K. and Ismail, Leila, A Multi-Dimensional Evaluation of Synthetic Data Generators, _IEEE Access_ , 2022, volume 10, pages 11147-11158, https://doi.org/10.1109/ACCESS.2022.3144765
* [73] Espinosa, Erica, and Alvaro Figueira, On the Quality of Synthetic Generated Tabular Data, _Mathematics_ , 2023, volume 11, issue 15, pages 3278, https://doi.org/10.3390/math11153278
* [74] Mendelevitch, Ofer, and Michael D. Lesh Fidelity and privacy of synthetic medical data, _arXiv preprint_ , 2021, https://doi.org/10.48550/arXiv.2101.08658