# Thinking Upstream: Ethics and Policy Opportunities in AI Supply Chains

David Gray Widder  Carnegie Mellon UniversityPittsburghPAUSA and  Richmond Y.
Wong  Georgia TechAtlantaGAUSA

(2018)

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock,
NY††booktitle: Woodstock ’18: ACM Symposium on Neural Gaze Detection, June
03–05, 2018, Woodstock, NY††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Looking upstream

After children were pictured sewing its running shoes in the early 1990s, Nike
at first disavowed the “working conditions in its suppliers’ factories”,
before public pressure led them to take responsibility for ethics in their
upstream supply chain (Doorey, 2011). In 2023, OpenAI responded to criticism
that Kenyan workers were paid less than $2 per hour to filter traumatic
content from its ChatGPT model by stating in part that it had outsourced the
work to a subcontractor, who managed workers’ payment and mental health
concerns (202, 2023).

In this position paper, we argue that policy interventions for AI Ethics must
consider AI as a supply chain problem, given how the political economy and
intra-firm relations structure AI production, in particular examining
opportunities upstream.

Much like physical goods, software is assembled from components developed by
many people across diverse contexts, in a “supply chain”. Widder and Nafus
suggested that “thinking about ethics and responsibility as chains of
relations surfaces specific locations in which ethical decision-making can
take place” (Widder and Nafus, 2022). They show how interviewees at dow the AI
supply chain view their ethical responsibility differently: developers“high”
in the chain saw what they were making as too “general purpose” to warrant
ethical scrutiny, but those downstream felt unequipped to remediate “ethical
debt” accrued upstream. By analogy to technical debt, “ethical debt” (Fiesler
and Garrett, 2020) refers to unaddressed flaws that may cause harm downstream,
to be “paid” by developers or users who later interact with the system.

However, AI ethics approaches often focus on the component being developed or
its downstream effects, rather than its upstream supply chain. Company AI
Ethics policy statements often scrutinize design while avoiding scrutiny of
downstream business uses (Greene et al., 2019). Ethical design principles and
checklists are insufficient— satire shows that systems satisfying “Fair”,
“Accountable”, “Transparent” design principles can nonetheless be patently
unethical when used for harm (Keyes et al., 2019), and others demonstrate that
in cases such as Deepfakes, harm is inherent in how a system is freely
distributed and thus widely used, not how it is designed (Widder et al.,
2022).

Relatedly, efforts like “Privacy by Design” and “Data Protection by Design”
are legal (in some jurisdictions) and engineering requirements that embed
privacy concerns throughout the development lifecycle for a particular
software component (Wong and Mulligan, 2019). However, a supply chain frame
would consider how that component depends on privacy assumptions or
affordances from its dependencies developed earlier in the supply chain, and
in turn for its downstream users. Gürses and Hoboken noted that privacy
research and policy interventions focus on sites of “technology consumption”
lower in the supply chain, disregarding modern and drastic changes in software
production, upstream in the supply chain (Gürses and van Hoboken, 2018).

Ethical design interventions for AI often think downstream, often drawing on
design futuring (Ballard et al., 2019; Wong, 2021; Fiesler, 2018; Martelaro
and Ju, 2020), scenarios (Zevenbergen et al., 2020), or value sensitive design
techniques (Shen et al., 2021) to consider how stakeholder harms might occur
during the deployment and use of AI systems. While useful, we argue that there
are unexplored opportunities for acting upstream.

##  2\. Acting upstream

Conceiving of AI ethics as a supply chain problem, and then looking up the
chain, surfaces “values levers”—practices that can pry open discussion about
values and ethics (Shilton, 2013) — that present opportunities for policy,
design and activism.

AI Supply Chains and Human Rights Law. Fukuda-Parr and Gibbons argue that
government and civil society must act to ground AI ethics in human rights
frameworks, but note that company AI ethics guidelines misuse “‘human rights’
as a rhetorical device” in the “absence of enforceable standards” and
regulations (Fukuda-Parr and Gibbons, 2021). Looking beyond AI supply chains
to those of physical goods, such standards and regulations exist. For example,
the United States banned “any goods, wares, articles, and merchandise mined,
produced, or manufactured wholly or in part” (emphasis added) in Xinjiang,
unless companies can prove they were not produced using Uyghur forced labor
(Flacks and Songy, 1200), and the UK Modern Slavery and Human Trafficking Act
requires companies to prevent forced labor in their supply chain (Kriebitz and
Max, 2020). Considering the working conditions of upstream AI data workers,
such as low paid annotators (202, 2023), suggests opportunities to apply human
rights law to workers in the AI supply chain.

Market-Based Policy Interventions: Disclosures, Procurement, and Choosy
Customers. Policy interventions focused on making producers of AI systems
disclose information about their upstream practices may create market
pressures to address ethical issues. A growing number of institutional
investors have expressed interest in investing in companies that meet
particular social or ethical standards, often termed “environmental and social
good” (ESG) (Schreck, 2013; Kaissar, 2022). While much ESG interest originates
in sustainability, monitoring agencies have begun to attend to companies’
possible digital harms, including labor rights, data privacy, and security
(Initiative, ). This disclosure is often voluntary and thus non-standardized,
but future policy initiatives may explore standardized and compulsory ESG
disclosures, incorporating AI supply chain ethics topics. Existing regulatory
agencies that require companies to make public disclosures about their
business practices, such as the U.S. Securities and Exchange Commission, may
play a role here (Wong et al., 2023).

Widder and Nafus explore the power of being a “choosy” customer, suggesting
that those sourcing software ‘might routinize asking suppliers for model
cards, if the data it was trained on was properly consented, if crowd workers
labeling the data were paid an appropriate wage”, as is often done in supply
chains for physical goods (Widder and Nafus, 2022). While not all customers
are choosy, governments can be perhaps more easily required to be so,
especially important in cases where governmental uses of AI affect people’s
freedoms and life chances. This may also make this scrutiny more routine and
thereby normalized. Writing about how government procurement and adoption of
ML systems are policy decisions as well as technical ones, Mulligan and
Bamberger advocate for using the power of procurement to require suppliers to
utilize “contestable design” which “exposes value-laden features” (Mulligan
and Bamberger, 2019), which might look like making it possible to challenge
upstream software features before their downstream consequences are ‘baked
in’. Disclosure-based approaches have drawbacks: Gansky and Mcdonald critique
“metadata maximalism”, questioning whether model cards and other “provenance
and trajectory documentation [provided] in order to enable transparency” can
“steer supplier practice via the discipline of the market” (Gansky and
McDonald, 2022), noting inherently messy AI supply chains.

Design and Activism in the Supply Chain. Design and activist practices can
make use of an upstream perspective by creating artifacts and tools that help
stakeholders understand, question, and advocate for changes upstream in the AI
supply chain. The Algorithmic Equity Toolkit (AEKit) was designed to help
citizens and community groups “find out more about a specific automated
decision system”, providing questions for advocates to ask policymakers and
technology vendors (Krafft et al., 2021), including upstream questions like
where a system’s data came from, who gathered it, with what tools, and for
what purpose. Artistic approaches to understanding AI systems, such as
Crawford and Joler’s “Anatomy of an AI System” can help surface and publicize
supply chain relationships in AI systems for further questioning (Crawford and
Joler, 2018).

Ethical Licensing. Novel software licenses are increasingly proposed and
experimented with in open source projects (Contractor et al., 2022; Carlos
Munoz Ferrandis, 2022; Declaration, 2018), which recognize harms from making
powerful AI freely available (Widder et al., 2022). As opposed to permissive
licenses, these require downstream users to think about their upstream
dependencies, and the ethical commitments they demand, and how these fit with
their own.

In sum, these ways of acting “upstream” present future opportunities for
design and policy interventions to address questions of AI Ethics.

## References

  * (1)
  * 202 (2023) 2023\.  Exclusive: The $2 Per Hour Workers Who Made ChatGPT Safer.  https://time.com/6247678/openai-chatgpt-kenya-workers/.
  * Ballard et al. (2019) Stephanie Ballard, Karen M. Chappell, and Kristen Kennedy. 2019.  Judgment Call the Game: Using value sensitive design and design fiction to surface ethical concerns related to technology. In _Proceedings of the 2019 on Designing Interactive Systems Conference - DIS ’19_. ACM Press, New York, New York, USA, 421–433.  https://doi.org/10.1145/3322276.3323697
  * Carlos Munoz Ferrandis (2022) Carlos Munoz Ferrandis. 2022\.  OpenRAIL: Towards Open and Responsible AI Licensing Frameworks.  https://huggingface.co/blog/open_rail.
  * Contractor et al. (2022) Danish Contractor, Daniel McDuff, Julia Katherine Haines, Jenny Lee, Christopher Hines, Brent Hecht, Nicholas Vincent, and Hanlin Li. 2022\.  Behavioral use licensing for responsible AI. In _2022 ACM Conference on Fairness, Accountability, and Transparency_. 778–788.
  * Crawford and Joler (2018) Kate Crawford and Vladan Joler. 2018.  Anatomy of an AI System.  https://anatomyof.ai/
  * Declaration (2018) Montreal Declaration. 2018\.  The Montreal Declaration for the Responsible Development of Artificial Intelligence.
  * Doorey (2011) David J. Doorey. 2011\.  The Transparent Supply Chain: From Resistance to Implementation at Nike and Levi-Strauss.  _Journal of Business Ethics_ 103, 4 (Nov. 2011), 587–603.  https://doi.org/10.1007/s10551-011-0882-1
  * Fiesler (2018) Casey Fiesler. 2018\.  Black Mirror, Light Mirror: Teaching Technology Ethics Through Speculation.  https://howwegettonext.com/the-black-mirror-writers-room-teaching-technology-ethics-through-speculation-f1a9e2deccf4
  * Fiesler and Garrett (2020) Casey Fiesler and Natalie Garrett. 2020.  Ethical Tech Starts with Addressing Ethical Debt.  (2020).
  * Flacks and Songy (1200) Marti Flacks and Madeleine Songy. Mon, 06/27/2022 - 12:00.  The Uyghur Forced Labor Prevention Act Goes into Effect.  (Mon, 06/27/2022 - 12:00).
  * Fukuda-Parr and Gibbons (2021) Sakiko Fukuda-Parr and Elizabeth Gibbons. 2021.  Emerging Consensus on ‘Ethical AI’: Human Rights Critique of Stakeholder Guidelines.  _Global Policy_ 12, S6 (2021), 32–44.  https://doi.org/10.1111/1758-5899.12965
  * Gansky and McDonald (2022) Ben Gansky and Sean McDonald. 2022.  CounterFAccTual: How FAccT undermines its organizing principles. In _2022 ACM Conference on Fairness, Accountability, and Transparency_. 1982–1992.
  * Greene et al. (2019) Daniel Greene, Anna Lauren Hoffmann, and Luke Stark. 2019\.  Better, nicer, clearer, fairer: A critical assessment of the movement for ethical artificial intelligence and machine learning. In _52nd Hawaii international conference on system sciences_.
  * Gürses and van Hoboken (2018) Seda Gürses and Joris van Hoboken. 2018.  _Privacy after the Agile Turn_.  Cambridge, 579–601.  https://doi.org/10.1017/9781316831960.032
  * Initiative (0) Global Reporting Initiative. 0\.  GRI Standards - English.  https://www.globalreporting.org/how-to-use-the-gri-standards/gri-standards-english-language/ [Online; accessed 2022-07-09].
  * Kaissar (2022) Nir Kaissar. 2022\.  Institutional Investors Are Flexing Their ESG Muscles.  https://www.bloomberg.com/opinion/articles/2022-04-13/institutional-investors-are-flexing-their-esg-muscles [Online; accessed 2022-07-09].
  * Keyes et al. (2019) Os Keyes, Jevan Hutson, and Meredith Durbin. 2019.  A mulching proposal: Analysing and improving an algorithmic system for turning the elderly into high-nutrient slurry. In _Extended Abstracts Of ACM Conference On Human Factors In Computing Systems_. 1–11.
  * Krafft et al. (2021) P. M. Krafft, Meg Young, Michael Katell, Jennifer E. Lee, Shankar Narayan, Micah Epstein, Dharma Dailey, Bernease Herman, Aaron Tam, Vivian Guetler, Corinne Bintz, Daniella Raz, Pa Ousman Jobe, Franziska Putz, Brian Robick, and Bissan Barghouti. 2021\.  An Action-Oriented AI Policy Toolkit for Technology Audits by Community Advocates and Activists. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ (Virtual Event, Canada) _(FAccT ’21)_. Association for Computing Machinery, New York, NY, USA, 772–781.  https://doi.org/10.1145/3442188.3445938
  * Kriebitz and Max (2020) Alexander Kriebitz and Raphael Max. 2020.  The Xinjiang Case and Its Implications from a Business Ethics Perspective.  _Human Rights Review_ 21, 3 (Sept. 2020), 243–265.  https://doi.org/10.1007/s12142-020-00591-0
  * Martelaro and Ju (2020) Nikolas Martelaro and Wendy Ju. 2020.  What could go wrong? Exploring the downsides of autonomous vehicles. In _12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications_. 99–101.
  * Mulligan and Bamberger (2019) Deirdre K Mulligan and Kenneth A Bamberger. 2019.  Procurement as policy: Administrative process for machine learning.  _Berkeley Tech. LJ_ 34 (2019), 773.
  * Schreck (2013) Philipp Schreck. 2013\.  Disclosure (CSR Reporting).  In _Encyclopedia of Corporate Social Responsibility_ , Samuel O Idowu, Nicholas Capaldi, Liangrong Zu, and Das Ananda Gupta (Eds.). Springer Berlin Heidelberg.  https://doi.org/10.1007/978-3-642-28036-8
  * Shen et al. (2021) Hong Shen, Wesley H. Deng, Aditi Chattopadhyay, Zhiwei Steven Wu, Xu Wang, and Haiyi Zhu. 2021\.  Value Cards: An Educational Toolkit for Teaching Social Impacts of Machine Learning through Deliberation. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. ACM, Virtual Event Canada, 850–861.  https://doi.org/10.1145/3442188.3445971
  * Shilton (2013) Katie Shilton. 2013\.  Values Levers: Building Ethics into Design.  _Science, Technology, & Human Values_ 38, 3 (2013), 374–397.  https://doi.org/10.1177/0162243912436985
  * Widder and Nafus (2022) David Gray Widder and Dawn Nafus. 2022.  Dislocated Accountabilities in the AI Supply Chain: Modularity and Developers’ Notions of Responsibility.  (Sept. 2022).  https://doi.org/10.48550/arXiv.2209.09780
  * Widder et al. (2022) David Gray Widder, Dawn Nafus, Laura Dabbish, and James Herbsleb. 2022.  Limits and Possibilities for “Ethical AI” in Open Source: A Study of Deepfakes. In _conference on fairness, accountability, and transparency_.
  * Wong (2021) Richmond Y Wong. 2021\.  Tactics of Soft Resistance in User Experience Professionals’ Values Work.  _Conference on Human-Computer Interaction_ 5, CSCW2 (2021), 1–28.
  * Wong et al. (2023) Richmond Y Wong, Andrew Chong, and R Cooper Aspegren. 2023\.  Privacy Legislation as Business Risks: How GDPR and CCPA are Represented in Technology Companies’ Investment Risk Disclosures.  _Proceedings of the ACM on Human-Computer Interaction_ 7, CSCW1 (2023).
  * Wong and Mulligan (2019) Richmond Y. Wong and Deirdre K. Mulligan. 2019.  Bringing Design to the Privacy Table: Broadening ”Design” in ”Privacy by Design” Through the Lens of HCI. In _CHI Conference on Human Factors in Computing Systems (CHI 2019)_.  https://doi.org/10.1145/3290605.3300492
  * Zevenbergen et al. (2020) Ben Zevenbergen, Allison Woodruff, and Patrick Gage Kelley. 2020\.  Explainability Case Studies.  https://doi.org/10.48550/ARXIV.2009.00246


# MMM and MMMSynth: Clustering of heterogeneous tabular data, and synthetic
data generation

Chandrani Kumari  Rahul Siddharthan

(30 October 2023)

###### Abstract

We provide new algorithms for two tasks relating to heterogeneous tabular
datasets: clustering, and synthetic data generation. Tabular datasets
typically consist of heterogeneous data types (numerical, ordinal,
categorical) in columns, but may also have hidden cluster structure in their
rows: for example, they may be drawn from heterogeneous (geographical,
socioeconomic, methodological) sources, such that the outcome variable they
describe (such as the presence of a disease) may depend not only on the other
variables but on the cluster context. Moreover, sharing of biomedical data is
often hindered by patient confidentiality laws, and there is current interest
in algorithms to generate synthetic tabular data from real data, for example
via deep learning.

We demonstrate a novel EM-based clustering algorithm, MMM (“Madras Mixture
Model”), that outperforms standard algorithms in determining clusters in
synthetic heterogeneous data, and recovers structure in real data. Based on
this, we demonstrate a synthetic tabular data generation algorithm, MMMsynth,
that pre-clusters the input data, and generates cluster-wise synthetic data
assuming cluster- specific data distributions for the input columns. We
benchmark this algorithm by testing the performance of standard ML algorithms
when they are trained on synthetic data and tested on real published datasets.
Our synthetic data generation algorithm outperforms other literature tabular-
data generators, and approaches the performance of training purely with real
data.

##  1 Introduction

Tabular datasets, consisting of heterogeneous variable types (categorical,
ordinal, numeric), are ubiquitous in data science and, in particular, in
biomedical research. Such datasets may additionally be heterogeneous in rows:
they may consist of a mix of different subtypes or categories, corresponding
to hidden structures not originally measured or are part of the dataset (such
as geographical location, socioeconomic class, genotype, etc).

For machine learning applications, often one variable (often called a
“response” or “output” variable) is of clinical interest (such as presence or
absence of a disease or disorder such as diabetes, birth weight of a fetus)
and the goal is to train a model to predict it from the other measurable
variables (often called “predictor ” or “input” variables). Patient
confidentiality often restricts the ability to share such datasets freely, and
several algorithms have been developed [1, 2, 3] to generate synthetic
datasets that closely resemble real datasets and can be used to train ML
models and shared freely.

Here we address both tasks, of clustering heterogeneous data, and of
generating realistic synthetic datasets as measured by their performance in
training models on them for ML prediction on the real data.

Several standard clustering algorithms exist for multidimensional data and are
implemented in machine learning packages such as scikit-learn [4], ClusterR[5]
and Clustering.jl[6]. These include agglomerative (hierarchical) clustering
methods such as UPGMA, partitioning algorithms such as $K$-means that seek to
optimize a particular distance metric across $K$ clusters, and mixture models
such as Gaussian Mixture Models (GMMs) that assume that the data is drawn from
a mixture of distributions and simultaneously learn the parameters of the
distributions and the assignment of data points to distributions.

These methods assume an underlying appropriate distance metric (such as
Euclidean distance) (agglomerative clustering, or $K$-means), or assume an
underlying probability distribution for the data (GMMs) which is to be
learned.

Many datasets consist of a mixture of binary (eg, gender) categorical (eg,
ethnicity), ordinal (eg, number of children), and numerical data (eg, height,
weight). Columns in such a tabular dataset may be correlated or
interdependent.

Here we propose an algorithm, which we call the Madras Mixture Model (MMM), to
cluster tabular data where the columns are assumed independently drawn from
either categorical or real data. For each clustering, we optimize the
likelihood of the total data being drawn from that clustering:

$P(D|K)=\prod_{j=1}^{K}P(d:d\in j)$ (1)

where $K$ is the number of clusters, and the right hand side is a product over
clusters for the likelihood that the subset of rows $d$ that are assigned to
cluster $j$ would be co-clustered (this is stated more precisely in Methods).

We assume that categorical variables are drawn from an unknown categorical
distribution with a Dirichlet prior, and numeric variables are drawn from an
unknown normal distribution with a normal-Gamma prior. Unlike with standard
GMM algorithms, we do not attempt to estimate the parameters of the
distributions, but integrate over them to obtain the likelihoods (see
Methods). Our clustering approach is a variation of expectation maximization
(EM) where, essentially, the M-step in the usual GMM algorithm is replaced
with this integration.

To determine the true number of clusters, we use the marginal likelihood (ML)
(sometimes called the Bayesian Occam’s razor) [7]. While the Bayesian
information criterion[8], is widely used as an approximation to the ML, its
performance on our synthetic dataset benchmarks was inferior.

The most accurate numerical calculation of the ML is using thermodynamic
integration (TI) [9, 10], reviewed in Methods, and we provide an
implementation using TI. This is computationally expensive since it involves
sampling at multiple different “temperatures” and integrating. As a faster
alternative, the ML is frequently estimated using the harmonic mean (HM) of
samples[11], but this is known to give a biased estimate in practice[12]. We
give an improved approximation, which we call HM$\beta$, involving a
fictitious inverse temperature $\beta$, which, for suitable $\beta$, converges
to the true value much faster than the HM on small datasets where the exact
answer is calculable, and also converges rapidly to a fixed value on larger
datasets. We demonstrate that HM$\beta$ with $\beta\approx 0.5$ produces
results comparable to TI on our synthetic datasets.

Finally, we use MMM as a basis for a synthetic tabular data generation
algorithm, MMMsynth. Each cluster is replaced with a synthetic cluster which
column-wise has the same statistical properties for the input variable, and
whose output variable is estimated with a noisy linear function learned from
the corresponding cluster in the true data. All these synthetic clusters are
then pooled to generate a synthetic dataset.

We assess quality of synthetic data by the performance, in predicting on real
data, of ML models trained on synthetic data. We demonstrate that this rather
simple approach significantly outperforms other published methods CT-GAN and
CGAN, and performs comparably with or better than Gaussian Copula and TVAE.
Our performance in many cases approaches the quality of prediction from
training on real data.

##  2 Methods

###  2.1 MMM: Clustering of heterogeneous data

Consider a tabular data set consisting of $L$ heterogeneous columns and $N$
rows. Each row of the set then consists of variables $x_{i}$,
$i=1,2,\ldots,L$. Each $x_{i}$ can be binary, categorical, ordinal, integer,
or real. We consider only categorical (including binary) or numeric data;
ordinal or numeric integer data can be treated as either categorical or
numeric depending on context. If there are missing data, they should first be
interpolated or imputed via a suitable method.

###  2.2 Discrete data, Dirichlet prior

For a categorical variable with $k$ values, the Dirichlet prior is
$P_{(}p)\propto\prod_{i=1}^{k}p_{i}^{c_{i}-1}$. For binary variables ($k=2$)
this is called the beta prior. If we have already observed data $D$ consisting
of $N$ observations, with each outcome $j$ occurring $N_{j}$ times, the
posterior predictive for outcome $x=i$ ($1\leq i,j\leq k$) is

$P(x=i|D)=\frac{N_{i}+c_{i}}{N+C}$ (2)

where $C=\sum_{i=1}^{k}c_{i}$.

###  2.3 Continuous data, normal-gamma prior

For a continuous normally-distributed variable, we use a normal-gamma prior,
as described in [13]. with four hyperparameters, which we call
$\mu_{0},\beta_{0},a_{0},b_{0}$:

$\displaystyle p(\mu,\lambda)$
$\displaystyle=\mathcal{N}\left(\mu|\mu_{0},(\beta_{0}\lambda)^{-1}\right)\mathrm{Gam}(\lambda|a_{0},b_{0})$
(3)
$\displaystyle=\frac{(\beta_{0}\lambda)^{1/2}}{\sqrt{2\pi}}e^{-\frac{\beta_{0}\lambda}{2}(\mu-\mu_{0})^{2})}\frac{1}{\Gamma(a_{0})}b_{0}^{a_{0}}\lambda^{a_{0}-1}e^{-b_{0}\lambda}$
$\displaystyle=\left(\frac{\beta_{0}}{2\pi}\right)^{1/2}\frac{b_{0}^{a_{0}}}{\Gamma(a_{0})}\lambda^{a_{0}-1/2}\exp\left(-\frac{\lambda}{2}\left[\beta_{0}(\mu-\mu_{0})^{2}+2b_{0}\right]\right).$
(4)

Here $\lambda$ is the inverse of the variance, $\lambda=\frac{1}{\sigma^{2}}$.

Given data $D$ consisting of $n$ items $x_{i}$, $i=1\ldots n$, the posterior
is

$\displaystyle p(\mu,\lambda|D)$
$\displaystyle=NG(\mu,\lambda|\mu_{n},\beta_{n},a_{n},b_{n})$
$\displaystyle=\left(\frac{\beta_{n}}{2\pi}\right)^{1/2}\frac{b_{n}^{a_{n}}}{\Gamma(a_{n})}\lambda^{a_{n}-1/2}\exp\left(-\frac{\lambda}{2}\left[\beta_{n}(\mu-\mu_{0})^{2}+2b_{n}\right]\right)$
(5)

where

$\displaystyle\mu_{n}$
$\displaystyle=\frac{\beta_{0}\mu_{0}+n\bar{x}}{\beta_{0}+n}$ (6)
$\displaystyle\beta_{n}$ $\displaystyle=\beta_{0}+n$ (7) $\displaystyle a_{n}$
$\displaystyle=a_{0}+\frac{n}{2}$ (8) $\displaystyle b_{n}$
$\displaystyle=b_{0}+\frac{1}{2}\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}+\frac{\beta_{0}n(\bar{x}-\mu_{0})^{2}}{2(\beta_{0}+n)}$
(9)

The posterior predictive, for seeing a single new data item $x$ given the
previous data $D$, is [13]

$p(x|D)=\pi^{-1/2}\frac{\Gamma(a_{n}+\frac{1}{2})}{\Gamma(a_{n})}\left(\frac{\Lambda}{2a_{n}}\right)^{\frac{1}{2}}\left(1+\frac{\Lambda(x-\mu_{n})^{2}}{2a_{n}}\right)^{-\left(a_{n}+\frac{1}{2}\right)}$
(10)

where

$\Lambda=\frac{a_{n}\beta_{n}}{b_{n}(\beta_{n}+1)}$ (11)

In log space:

$\log
p(x|D)=-0.5\log\pi+\log\Gamma(a_{n}+\frac{1}{2})-\log\Gamma(a_{n})+0.5(\log\Lambda-\log(2a_{n}))-\left(a_{n}+\frac{1}{2}\right)\log\left(1+\frac{\Lambda(x-\mu_{n})^{2}}{2a_{n}}\right)$
(12)

The marginal likelihood is

$p(D)=\frac{\Gamma(a_{n})}{\Gamma(a_{0})}\frac{b_{0}^{a_{0}}}{b_{n}^{a_{n}}}\left(\frac{\beta_{0}}{\beta_{n}}\right)^{1/2}(2\pi)^{-n/2}$
(13)

###  2.4 Optimizing likelihood of a clustering by expectation maximization

Let the data $D$ consist of $N$ rows, so that $D_{i}$ is the $i$’th row. Let
the model be denoted by $M_{K}$ where $K$ is the number of clusters. Each
cluster has its own parameters of the categorical or normal distribution for
each column which we call $\vec{\theta}$, with $\vec{\theta}_{\ell j}$ being
the vector of parameters for column $\ell$ in cluster $j$. The vector has
$k-1$ independent components for a categorical distribution of $k$ categories,
and $2$ components for a normal distribution, which are all continuous.
Another, discrete parameter for the model is the detailed cluster assignment
of each row $D_{i}$ to each cluster $C_{j}$. This can be described by a vector
$\vec{A}$ of length $N$, whose elements $A_{i}$ take values from $1$ to $K$. A
specific clustering is described by
$\vec{\Theta}=\left\\{\vec{\theta},\vec{A}\right\\}$.

Given $K$, we seek an optimal clustering (i.e, optimal choice of $\vec{A}$) by
maximum likelihood. The likelihood being maximized is

$\displaystyle\vec{A}^{*}$
$\displaystyle=\operatorname*{arg{}max}_{\vec{A}}\prod_{j=1}^{K}P(d_{j})$
$\displaystyle=\operatorname*{arg{}max}_{\vec{A}}\prod_{j=1}^{K}\int
P(d_{j}|\vec{\theta_{j}})P(\vec{\theta_{j}})d\vec{\theta_{j}}.$ (14)

Here $d_{j}$ is shorthand for $\\{D_{i}|A_{i}=j\\}$, that is, it is the set of
rows from $D$ that belong to cluster $j$, and $\vec{\theta}_{j}$ is the set of
parameters $\vec{\theta}$ specific to cluster $j$. The unknown parameters
$\vec{\theta_{j}}$ are integrated over, separately for each cluster, and
$P(\vec{\theta_{j}})$ is a Dirichlet or normal-Gamma prior as appropriate.
That is, we seek to find that assignment of individual rows to clusters,
$\vec{A}$, such that the product of the likelihoods that the set of rows
$D_{j}$ that have been assigned to the cluster $j$ are described by the same
probabilistic model is maximized. There is an implicit product over columns,
which are assumed independent.

In contrast to EM implementations of GMMs, we only seek to learn $\vec{A}$; we
do not seek the parameters of the probabilistic model $\vec{\theta}$, but
integrate over them.

An algorithm for clustering into $K$ clusters could be

  1. 1.

Initialize with a random assignment of rows to clusters.

  2. 2.

Calculate a score matrix $L_{ij}$. For each row $i$, this is the likelihood
that $D_{i}$ belongs to cluster $j$ for all $j$ (excluding $i$ from its
current cluster).

  3. 3.

Assign each row $i$ to the cluster corresponding to that value of $j$ which
maximises $L_{ij}$. (This is similar to the E-step in EM.)

  4. 4.

Repeat from step 2 until no reassignments are made.

Instead of the M-step in EM, step 2 calculates likelihoods according to the
posterior predictives for the categorical (2) and normal (10) distributions.
The likelihood for a row is the product of the likelihoods over columns. We
work with log likelihoods.

In our implementation, we iterate starting from one cluster. After optimizing
each $K$ clustering, starting at $K=1$, we use a heuristic to initialize $K+1$
clusters: we pick the poorest-fitting $\frac{N}{K+1}$ rows (measured by their
posterior predictive for the cluster that they are currently in) and move them
to a new cluster, and then run the algorithm as above.

We can choose to either stop at a pre-defined $K$, or identify the optimal $K$
via marginal likelihood, as described below (the optimal $K$ could be 1).

###  2.5 Identifying the correct $K$: Marginal likelihood

Equation 14 maximizes the likelihood of a clustering over $\vec{A}$, while
marginalizing over $\vec{\theta}$. The correct $K$ in the Bayesian approach is
the $K$ that maximises the marginal likelihood (ML) marginalized over _all_
parameters including $\vec{A}$. In other words, while one can increase the
likelihood in eq 14 by splitting into more and more clusters, beyond a point
this will lead to overfitting; the full ML penalizes this (an approach also
called the Bayesian Occam razor [7]).

Unfortunately, exact calculation of the ML (marginalizing over $\vec{A}$) is
impossible. There are several approaches using sampling; we review two below
(harmonic mean [HM], and thermodynamic integration [TI]), before introducing
our own, a variant of HM which we call HM$\beta$, which we show is more
accurate than HM, and on our data, comparably accurate to TI while being
faster.

####  2.5.1 Arithmetic Mean and Harmonic Mean

A straightforward estimation of the marginal likelihood for $K$ clusters
($ML_{K}$) would be to sample uniformly from the parameter space for
$\vec{A}$, and calculate the average likelihood over $M$ samples (there is an
implicit marginalization over $\vec{\theta}$ throughout):

$ML_{K}\equiv P(D|K)\approx\frac{1}{M}\sum_{m=1}^{M}P(D|K,\vec{A}_{m})$ (15)

This is the arithmetic mean (AM) estimate, and tends to be biased to lower
likelihoods because the region of high-likelihood parameters is very small.

An alternative is to start from Bayes’ theorem:

$P(\vec{A}|D,K)=\frac{P(D|\vec{A},K)P(\vec{A}|K)}{\sum_{\vec{A^{\prime}}}P(D|\vec{A^{\prime}},K)P(\vec{A^{\prime}}|k)}$
(16)

The denominator on the right is the marginal likelihood for $K$ clusters.
Rearranging,

$\frac{P(\vec{A}|K)}{ML_{K}}=\frac{P(\vec{A}|D,K)}{P(D|\vec{A},K)}$ (17)

and summing over $\vec{A}$ with $\sum_{\vec{A}}P(\vec{A}|K)=1$,

$\frac{1}{ML_{K}}=\sum_{\vec{A}}\frac{P(\vec{A}|D,K)}{P(D|\vec{A},K)}$ (18)

If we sample $\vec{A}$ from the distribution $P(\vec{A}|D,K)$, then for $M$
samples we have

$ML_{K}\approx\left(\frac{1}{M}\sum_{m=1}^{M}\frac{1}{P(D|\vec{A}_{m},K)}\right)^{-1}.$
(19)

This is the “harmonic mean” (HM) approximation. Both the HM and the AM can be
derived via different choices of an importance sampling distribution in a
Metropolis-Hastings scheme [10]. The HM is known to be biased towards higher
likelihoods in practice, oppositely to the AM.

####  2.5.2 Thermodynamic integration

Thermodynamic integration (TI), a technique borrowed from physics, was
described in the statistical inference context by Gelman and Meng [9]. The
following quick summary is adapted to our notation from Lartillot and Philippe
[10].

Suppose one has an un-normalized density in parameter space, parametrized by
$\beta$, $q_{\beta}(\vec{A})$. We can define a “partition function”

$Z_{\beta}=\int q_{\beta}(\vec{A})d\vec{A}.$ (20)

In our case $\vec{A}$ is discrete, so the integral, here and below, should be
interpreted as a sum. From this we get a normalized density

$p_{\beta}(\vec{A})=\frac{1}{Z_{\beta}}q_{\beta}(\vec{A}).$ (21)

We then have

$\displaystyle\frac{\partial}{\partial\beta}\log Z_{\beta}$
$\displaystyle=\frac{1}{Z_{\beta}}\frac{\partial Z_{\beta}}{\partial\beta}$
$\displaystyle=\frac{1}{Z_{\beta}}\frac{\partial}{\partial\beta}\int
q_{\beta}(\vec{A})d\vec{A}$
$\displaystyle=\int\frac{1}{q_{\beta}(\vec{A})}\frac{\partial
q_{\beta}(\vec{A})}{\partial\beta}\frac{q_{\beta}(\vec{A})}{Z_{\beta}}dA$
$\displaystyle=E_{\beta}\left[\frac{\partial\log
q_{\beta}(\vec{A})}{\partial\beta}\right]$

Defining $U(\theta)=\frac{\partial}{\partial\beta}\log q_{\beta}(\vec{A})$,
and integrating from $\beta=0$ to $1$,

$\log Z_{1}-\log Z_{0}=\int_{0}^{1}E_{\beta}[U]d\beta.$ (22)

Consider the particular choice

$q_{\beta}(\vec{A})=P(D|\vec{A},K)^{\beta}P(\vec{A}|K)$ (23)

where, as above, $K$ is the number of clusters. Then $q_{0}$ is the prior for
$\vec{\theta}$, and $q_{1}$ is proportional to the posterior. Therefore
$Z_{0}$ is 1 (since $q_{0}$ is normalized) and $Z_{1}$ is the marginal
likelihood. Substituting,

$\log\mathrm{ML}=\int_{0}^{1}E_{\beta}\left[\log P(D|\vec{A},K)\right]d\beta$
(24)

.

The expectation $E_{\beta}$ is calculated by sampling at various $\beta$ and
the integral is found by Simpson’s rule.

####  2.5.3 A faster approximation

We return to the HM approximation (eq 18). The problem is that the
distribution $P(\vec{A}|D,K)$ is strongly peaked around the optimal parameters
$\vec{A}$. To broaden the distribution we can introduce a fictitious inverse
temperature $\beta$ (not the same as in TI), and write

$\frac{1}{ML_{K}}=\sum_{\vec{A}}\frac{P(\vec{A}|D,K)^{\beta}P(\vec{A}|D,K)^{1-\beta}}{P(D|\vec{A},K)}$
(25)

But

$P(\vec{A}|D,K)=\frac{P(D|\vec{A},K)P(\vec{A}|K)}{ML_{K}}$ (26)

and this gives

$\frac{1}{ML_{K}}=\sum_{\vec{A}}P(\vec{A}|D,K)^{\beta}\left(\frac{P(\vec{A}|K)}{ML_{K}}\right)^{1-\beta}P(D|\vec{A},K)^{-\beta}$
(27)

and therefore

$ML_{K}^{-\beta}=\sum_{\vec{A}}P(\vec{A}|D,K)^{\beta}P(\vec{A}|K)^{1-\beta}P(D|\vec{A},K)^{-\beta}$
(28)

We can evaluate $ML_{K}$ from this by sampling from

$\tilde{P}(\vec{A}|D,K)\equiv P(\vec{A}|D,K)^{\beta}P(\vec{A}|K)^{1-\beta}$
(29)

(in practice from $P(D|\vec{A},K)^{\beta}$ which is proportional to this)
rather than $P(\vec{A}|D,K)$. However, this distribution is not normalized.
Let
$\sum_{\vec{A}}\tilde{P}(\vec{A}|D,K)=\sum_{\vec{A}}P(\vec{A}|D,K)^{\beta}P(\vec{A}|K)^{1-\beta}=Z\neq
1$. If we take $M$ samples from $\tilde{P}(\vec{A}|D,K)$, then

$\displaystyle ML_{K}^{-\beta}$
$\displaystyle=\frac{1}{M}\sum_{m=1\atop\mathrm{samples~{}from~{}}\tilde{P}}^{M}P(D|\vec{A},K)^{-\beta}\times
Z$ (30)
$\displaystyle\equiv\left<P(D|\vec{A},K)^{-\beta}\right>_{\tilde{P}}Z.$ (31)

To estimate $Z$ we are forced to sample from the normalized distribution
$P(\vec{A}|D,K)$:

$\displaystyle Z$
$\displaystyle=\sum_{\vec{A}}\tilde{P}(\vec{A}|D,K)=\sum_{\vec{A}}\frac{P(\vec{A}|D,K)^{\beta}P(\vec{A}|K)^{1-\beta}}{P(\vec{A}|D,K)}P(\vec{A}|D,K)$
(32)
$\displaystyle\equiv\left<\frac{P(\vec{A}|D,K)^{\beta}P(\vec{A}|K)^{1-\beta}}{P(\vec{A}|D,K)}\right>_{P}$
(33)
$\displaystyle=\left<\frac{P(D|\vec{A},K)^{\beta-1}P(\vec{A}|K)^{\beta-1}}{ML_{K}^{\beta-1}}P(\vec{A}|K)^{1-\beta}\right>_{P}$
(34)

Substituting,

$ML_{K}^{-\beta}=ML_{K}^{1-\beta}\left<P(D|\vec{A},K)^{-\beta}\right>_{\tilde{P}}\left<P(D|\vec{A},K)^{\beta-1}\right>_{P}$

$ML_{K}^{-1}=\left<P(D|\vec{A},K)^{-\beta}\right>_{\tilde{P}}\left<P(D|\vec{A},K)^{\beta-1}\right>_{P}$

and finally

$\log
ML_{K}=-\log\left<P(D|\vec{A},K)^{-\beta}\right>_{\tilde{P}}-\log\left<P(D|\vec{A},K)^{\beta-1}\right>_{P}$
(35)

where, again, $\left<\ldots\right>_{\tilde{P}}$ means an average over $M$
samples from $P(D|\vec{A},K)^{\beta}$, and $\left<\ldots\right>_{P}$ means an
average over $M$ samples from $P(\vec{A}D,K)$. This expression reduces to the
arithmetic mean if $\beta=0$ and to the harmonic mean if $\beta=1$. We refer
to it as HM$\beta$.

Figure 1: For a dataset of 20 rows, the marginal likelihood can be calculated
exactly (solid line); this is compared with TI and with HM$\beta$ at various
$\beta$. $\beta=0.5$ gives results close to the exact value.

We assess an optimal choice of $\beta$ using a small dataset of 20 rows where
the marginalization can be carried out explicitly, by summing over all
1,048,574 possible cluster assignments to two clusters (figure 1). The TI
method converges quite quickly to the exact answer, and the HM$\beta$ for
$\beta=0.5$ also gives good results. On larger datasets too we found
$\beta=0.5$ an optimal choice (as in next subsection), though the exact value
of the ML is not known.

####  2.5.4 Synthetic data for benchmarking clustering

We generated multiple datasets, each with 10 columns, 5000 rows, with five
clusters each, in a 5:4:3:2:1 ratio, with varying parameters, as follows. For
categorical data, columns in each cluster $j$ were sampled from vectors
$\boldsymbol{v}_{0}+\Delta\boldsymbol{v}_{j}$ where $\boldsymbol{v}_{0}$ was
common to all clusters and $\boldsymbol{v}_{j}$ was specific to the $j$’th
cluster, and $\Delta$ was varied from 0.5 to 4.5, with smaller values
indicating greater similarity among clusters. Here, five columns were binary
and five were 4-valued. For numeric data, the means in different clusters were
separated by 1.0 and the standard deviations were separated by $\delta\sigma$,
which varied from 0.5 to 4.5, with smaller $\delta\sigma$ indicating more
distinct clusters. We also simulated numeric data where the means in all
clusters were the same and only the standard deviations differed by
$\delta\sigma$. Finally, we generated mixed datasets, which included five
numeric and five categorical (4-valued) columns, varying the parameter
$\Delta$ as above and fixing $\delta\sigma=5.0-\Delta$, so that increasing
$\Delta$ implies increasing similarity among all columns.

###  2.6 Assessment of benchmarking

We used the adjusted Rand index (ARI) [14] to compare true and predicted
clusters.

We also used normalized clustering accuracy, as provided by the GenieClust
[15] suite, and defined by Gagolewski in [16], where applicable; the results
were similar, but this formula applies only when the true and predicted
numbers of clusters are the same.

###  2.7 MMMsynth: generating synthetic data with MMM

####  2.7.1 Synthetic data generation algorithm

MMMSynth uses MMM to pre-cluster an input dataset, excluding the output
column. Each cluster is assumed, as in MMM, to consist of independent columns
that are either categorical or numeric. The parameters of the corresponding
multinomial or Gaussian distribution are fitted to each column in each
cluster, and a new cluster of the same size is generated by sampling from
these distributions. A linear model is fitted to the output column in each
real cluster and used to generate the output column in the synthetic clusters.
The synthetic clusters are finally combined to produce a full dataset of the
same size as the original dataset.

For comparison we generated synthetic data using the methods availbale in
literature. We used synthetic data vault[17] libraries in python to generate
synthetic data using TVAE, Gaussian Copula, CTGAN and CGAN.

####  2.7.2 Benchmarking MMMSynth-generated synthetic datasets

To evaluate the similarity of the generated data of the real data, we trained
machine learning models (logistic regression, random forest) on the synthetic
data and evaluated their predictive performance on the real dataset. We also
compared the performance of a model trained on the real dataset in predicting
on the same dataset. We used six datasets from the UCI machine learning
repository.

###  2.8 Benchmarks: Real datasets used

We used the following datasets from the UCI Machine Learning Repository[18],
some of which were obtained via the Kaggle platform:

  * •

Abalone: predicting age of abalone from physical measurements, 8 predictors
(input variables), 1323 rows, from UCI
https://archive.ics.uci.edu/ml/datasets/abalone/

  * •

Heart failure prediction dataset: compiled from UCI Machine learning
Repository, 11 predictors, 918 rows, from
https://www.kaggle.com/fedsoriaon/heart-failure/prediction

  * •

Pima Indians diabetes: 8 predictors, 768 rows, from
https://www.kaggle.com/datasets/uciml/pima-indians-diabetes-database, source
UCI

  * •

Breast cancer Wisconsin (Diagnostic) dataset: 30 predictors, 569 rows, from
https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29

  * •

Maternal health risk data: 7 predictors, 676 rows, from
https://www.kaggle.com/datasets/csafrit2/maternal-health-risk-data

  * •

Stroke prediction dataset: 10 predictors, 4909 rows, of which 209 positive and
4700 negative, from https://www.kaggle.com/datasets/zzettrkalpakbal/full-
filled-brain-stroke-dataset

  * •

Connectionist Bench (Sonar, Mines vs. Rocks) dataset (60 predictors, 207 rows,
from
https://archive.ics.uci.edu/dataset/151/connectionist+bench+sonar+mines+vs+rocks

All of these have binary output variables. For clustering benchmarking, all
were used (sonar was used as part of the ClustBench set, below). For synthetic
data, the stroke dataset was omitted since it was highly unbalanced.

In addition, datasets from the UCI subdirectory of ClustBench [19] were used,
many of which have multi-valued outputs. These consist of sonar (described
above), as well as

  * •

ecoli: 7 predictors, 335 rows, 8 categories

  * •

glass: 9 predictiors, 213 rows, 6 categories

  * •

ionosphere: 34 predictors, 350 rows, 2 categories

  * •

statlog: 19 predictors, 2309 rows, 7 categories

  * •

wdbc: 30 predictors, 568 rows, 2 categories

  * •

wine: 13 predictors, 177 rows, 3 categories

  * •

yeast: 8 predictors, 1483 rows, 10 categories

##  3 Results

###  3.1 Clustering algorithm performance

####  3.1.1 Synthetic data: purely categorical, purely normal, mixed

We generate three kinds of synthetic datasets: purely categorical, purely
numeric (normally distributed), and mixed, as described in Methods. Each
dataset has five clusters. We vary a parameter ($\delta\sigma$ for numeric,
$D$ for categorical), as described in Methods, to tune how similar the
clusters are to each other: large $\delta\sigma$ or small $D$ will indicate
similar clusters and a harder problem. Figure 2 shows the results for
normalized accuracy: for purely normally distributed data MMM performs
comparably with Gaussian mixture models, for purely categorical data we
greatly outperform all methods, with only $K$-means with one-hot encoding
coming close, and for mixed normal+categorical data, too, our performance is
superior to other methods. We run MMM in three modes: telling it the true
cluster size, or using the TI and HM$\beta$ methods (with $\beta=0.5$). All
other methods are told the correct cluster size.

Figure 2: Clustering of four kinds of synthetic datasets: (A) purely numeric
(normally distributed, differing means and variances), (B) purely numeric
(normally distributed, same mean but differing variances), (C) purely
categorical, and (D) mixed.

####  3.1.2 Real data

We consider the UCI abalone, breast, diabetes, heart, MHRD and stroke datasets
described in Methods, which have binary outcome variables, and also eight
datasets from the UCI machine learning database which are included in
ClustBench [19], which have output labels of varying number from 2 to 10.
These are intended as tests of classification, not clustering, tasks.
Nevertheless, clustering these datasets on input variables (excluding the
outcome variable) often shows significant overlap with the clustering
according to the output variable, as shown in figure 3. This suggests that we
are recovering real underlying structure in the data. We compare various other
methods, all of which were run with the correct known number of clusters,
while MMM was run with (“fixed”) and without telling it the true number of
clusters.

Figure 3: Performance of MMM, using the TI criterion, and “MMM (fixed)” where
it is told the true number of clusters versus other programs, on eight
datasets from ClustBench (A) and six datasets from UCI (B). Performance is
reported using Adjusted Rand Index.

####  3.1.3 MMMSynth: ML performance on synthetic data

We use the six kaggle/UCI datasets (Abalone, Breast Cancer, Diabetes, Heart,
MHRD and Sonar) benchmarked above to generate synthetic data, as described in
Methods, train ML algorithms (logistic regression, random forest) on these,
and measure the predictive performance on the real data. Figures 4 show the
results. Also tested are Triplet-based Variational Autoencoder (TVAE),
Gaussian Copula (GC), Conditional Generative Adversarial Network (CTGAN),
Copula Generative Adversarial Network (CGAN). In all datasets, we show
performance comparable to training on real data. TVAE and GC also perform well
on most datasets, while CGAN and CTGAN show poorer performance. All programs
were run with default parameters.

Figure 4: Random forest (A) and logistic regression (B) models were trained on
the real data and on synthetic data generated using MMMSynth, TVAE, GC, CGAN
and CTGAN and their predictive performance evaluated on the real datasets. The
AUC (area under ROC curve) is shown for each method and each dataset.

##  4 Discussion

We present a clustering algorithm, MMM, that clusters heterogeneous data
consisting of categorical and numeric columns. We demonstrate good performance
on a variety of publicly available datasets and on our synthetic data
benchmarks. Speed optimizations will be explored in future work.

Currently the columns are assumed to be independent, but it will be a
straightforward exercise to use a multivariate Gaussian to describe the
numeric columns. This too will be explored in future. Despite this, our
performance is comparable to and sometimes better than scikit-learn’s GMM
implementation on the real-world UCI and clustbench data presented here.

We further use MMM as a basis for a synthetic data generation algorithm,
MMMSynth. We demonstrate that MMMSynth generates synthetic data that closely
resembles real datasets. Our method performs better than current synthetic
data generation algorithms in the literature (TVAE, Gaussian Copula, CGAN and
CTGAN). Notably, all these methods explicitly assume and model correlations
between input columns: CGAN and Gaussian Copula use copula functions to
capture correlations between variables, while TVAE and CTGAN employ deep
learning using all columns. In contrast, we first cluster the data and then
assume that, within each cluster, columns are uncorrelated.

Our approach indirectly accounts for some correlations: for example, if two
binary columns are correlated (1 tends to occur with 1, and 0 with 0), we
would be likely to cluster the 1’s together and 0’s together. It will also
account for multimodal numeric columns since these would be better represented
as a sum of Gaussians. In tests on synthetically-generated non-normally-
distributed numeric data (for example, Gamma-distributed with long tails), MMM
breaks the data into multiple clusters, suggesting an attempt to approximate
the Gamma distribution as a sum of Gaussians. This will be explored in a
future work. Nevertheless, we do not see such a proliferation of clusters when
running on real datasets.

##  5 Acknowledgements

This work grew from a project on personalized clinical predictions, for which
we acknowledge discussion and collaboration with Gautam Menon, Uma Ram,
Ponnusamy Saravanan, and particularly Leelavati Narlikar with whom we
extensively discussed this work and whose insights were invaluable. We also
acknowledge useful discussions with Durga Parkhi on synthetic data generation.

##  6 Funding

We acknowledge funding from BIRAC grant bt/ki-data0404/06/18 (RS), and the
IMSc Centre for Disease Modelling (ICDM) funded via an apex project at IMSc by
the Department of Atomic Energy, Government of India (CK, RS). The funders had
no role in the data collection, research, analysis, writing or submission of
the manuscript.

##  7 Author contributions

Conceptualization: CK, RS

Data Curation: CK, RS

Formal Analysis: CK, RS

Funding Acquisition: RS

Investigation: CK, RS

Methodology: CK, RS

Project Administration: RS

Resources: RS

Software: CK, RS

Supervision: RS

Validation: CK, RS

Visualization: CK, RS

Writing – Original Draft Preparation: CK, RS

Writing – Review & Editing: CK, RS

##  8 Code availability

MMM and MMMsynth are available on
https://github.com/rsidd120/MadrasMixtureModel under the MIT licence. They are
implemented in Julia.

##  9 Declaration of interests

The authors declare no competing interests.

## References

  * [1] Xu L, Skoularidou M, Cuesta-Infante A, Veeramachaneni K.  Modeling tabular data using conditional gan.  Advances in neural information processing systems. 2019;32.
  * [2] Li Z, Zhao Y, Fu J.  Sync: A copula based framework for generating synthetic data from aggregated sources.  In: 2020 International Conference on Data Mining Workshops (ICDMW). IEEE; 2020. p. 571–578.
  * [3] Xu L, et al.  Synthesizing tabular data using conditional GAN.  Massachusetts Institute of Technology; 2020.
  * [4] Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al.  Scikit-learn: Machine learning in Python.  the Journal of machine Learning research. 2011;12:2825–2830.
  * [5] Mouselimis L. ClusterR: Gaussian Mixture Models, K-Means, Mini-Batch-Kmeans, K-Medoids and Affinity Propagation Clustering; 2023.  Available from: https://CRAN.R-project.org/package=ClusterR.
  * [6] Stukalov A, Lin D.  Clustering. jl.  Julia Statistics Available online at: https://github com/JuliaStats/Clustering jl (accessed September 30, 2021). 2021;.
  * [7] MacKay DJ.  Bayesian interpolation.  Neural computation. 1992;4(3):415–447.
  * [8] Neath AA, Cavanaugh JE.  The Bayesian information criterion: background, derivation, and applications.  Wiley Interdisciplinary Reviews: Computational Statistics. 2012;4(2):199–203.
  * [9] Gelman A, Meng XL.  Simulating normalizing constants: From importance sampling to bridge sampling to path sampling.  Statistical science. 1998; p. 163–185.
  * [10] Lartillot N, Philippe H.  Computing Bayes factors using thermodynamic integration.  Systematic biology. 2006;55(2):195–207.
  * [11] Newton MA, Raftery AE.  Approximate Bayesian inference with the weighted likelihood bootstrap.  Journal of the Royal Statistical Society Series B: Statistical Methodology. 1994;56(1):3–26.
  * [12] Xie W, Lewis PO, Fan Y, Kuo L, Chen MH.  Improving marginal likelihood estimation for Bayesian phylogenetic model selection.  Systematic biology. 2011;60(2):150–160.
  * [13] Murphy KP. Conjugate Bayesian analysis of the Gaussian distribution;.  https://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf.
  * [14] Hubert L, Arabie P.  Comparing partitions.  Journal of classification. 1985;2:193–218.
  * [15] Gagolewski M.  genieclust: Fast and robust hierarchical clustering.  SoftwareX. 2021;15:100722.
  * [16] Gagolewski M.  Normalised clustering accuracy: An asymmetric external cluster validity measure.  arXiv preprint arXiv:220902935v2. 2022;.
  * [17] Patki N, Wedge R, Veeramachaneni K.  4700\.  In: IEEE International Conference on Data Science and Advanced Analytics (DSAA); 2016. p. 399–410.
  * [18] Dua D, Graff C, et al.  UCI machine learning repository. 2017;.
  * [19] Gagolewski M.  A framework for benchmarking clustering algorithms.  SoftwareX. 2022;20:101270.