# Domain-agnostic and Multi-level Evaluation of Generative Models

Girmaw Abebe Tadesse
IBM Research - Africa
<EMAIL_ADDRESS>
Jannis Born
IBM Research - Zurich
<EMAIL_ADDRESS>
Celia Cintas
IBM Research - Africa
<EMAIL_ADDRESS>
William Ogallo
IBM Research - Africa
<EMAIL_ADDRESS>
Dmitry Zubarev
IBM Research - Almaden
<EMAIL_ADDRESS>
Matteo Manica
IBM Research - Zurich
<EMAIL_ADDRESS>
Komminist Weldemariam
IBM Research - Yorktown Heights
<EMAIL_ADDRESS>
###### Abstract

While the capabilities of generative models heavily improved in different
domains (images, text, graphs, molecules, etc.), their evaluation metrics
largely remain based on simplified quantities or manual inspection with
limited practicality. To this end, we propose a framework for Multi-level
Performance Evaluation of Generative mOdels (MPEGO), which could be employed
across different domains. MPEGO aims to quantify generation performance
hierarchically, starting from a sub-feature-based low-level evaluation to a
global features-based high-level evaluation. MPEGO offers great
customizability as the employed features are entirely user-driven and can thus
be highly domain/problem-specific while being arbitrarily complex (e.g.,
outcomes of experimental procedures). We validate MPEGO using multiple
generative models across several datasets from the material discovery domain.
An ablation study is conducted to study the plausibility of intermediate steps
in MPEGO. Results demonstrate that MPEGO provides a flexible, user-driven, and
multi-level evaluation framework, with practical insights on the generation
quality. The framework, source code, and experiments will be available at:
https://github.com/GT4SD/mpego.

_K_ eywords Generative models $\cdot$ Evaluation $\cdot$ Data-centric AI
$\cdot$ Foundation models

##  1 Introduction

Machine Learning (ML) methods, particularly generative models, are effective
in addressing critical problems across different domains, which includes
material sciences. Examples include the design of novel molecules by combining
data-driven techniques and domain knowledge to efficiently search the space of
all plausible molecules and generate new and valid ones [1, 2, 3, 4].
Traditional high-throughput wet-lab experiments, physics-based simulations,
and bioinformatics tools for the molecular design process heavily depend on
human expertise. These processes require significant resource expenditure to
propose, synthesize and test new molecules, thereby limiting the exploration
space [5, 6, 7].

For example, generative models have been applied to facilitate the material
discovery process by employing inverse molecular design problem. This approach
transforms the conventional and slow discovery process by mapping the desired
set of properties to a set of structures. The generative process is then
optimized to encourage the generation of molecules with those selected
properties. Countless approaches have been suggested for such tasks, most
prominently VAEs with different sampling techniques [8, 9, 10]), GANs [11,
12], diffusion models [13], flow networks [14] and Transformers [15].

Though the generation capability has been tremendously improved recently, the
quantitative evaluation of these generative models in different domains
remains a grand challenge [16]. Some of the reasons include the multi-
objective nature of real discovery problems, the intricacy of evaluating
relevant features in-silico, and the lack of widely accepted domain- and
model-agnostic evaluation frameworks. As a result, existing benchmarks and
toolkits in material sciences, such as MOSES [17] or GuacaMol [18] include
limited metrics, such as validity and uniqueness, that lack the capacity to
evaluate the complex nature of the generation process (e.g., interactions of
multiple properties), thereby less effective to provide meaningful insights to
subject matter experts (SMEs) to facilitate practical impact.

In this paper, we introduce a Multi-level Performance Evaluation of Generative
mOdels (MPEGO) framework (see Fig. 1), which aims to hierarchically
characterize and quantify the capability of generative models, using material
discovery domain as a use case. To that end, MPEGO is a model- and domain-
agnostic framework, and its core design is derived from two main requirements:
representative examples (of training and generated samples) and one or
multiple features (extracted from these samples). Metrics derived from MPEGO
are also interpretable and provide multi-level abstractions of the generation
process. Specifically, the contributions of this paper are as follows:

  1. 1.

We provide a multi-level and domain-agnostic evaluation of generative models,
starting with sub-feature- or feature-based low-level evaluation to global
features based high level-evaluation.

  2. 2.

We devise a generation frequency analysis that aims to identify and
characterize subsets of samples generated with extreme frequencies.

  3. 3.

We validate MPEGO framework on multiple generative models (e.g., GCPN [19],
GraphAF [20], MolGX [21] and Regression Transformer [15], and multiple
datasets (e.g., ZINC-250K [22] and MOSES [17] and
CIRCA111https://circa.res.ibm.com/.

  4. 4.

We conduct ablation studies to analyze MPEGO’s sensitivity to design choices.

##  2 Related Work

The state-of-the-art evaluation approaches for generative models aim to
quantify pre-determined requirements, such as diversity and validity, using a
variety of metrics. Frechet ChemNet Distance (FCD) [23] is one of such
metrics, and it measures the distance between hidden representations drawn
from sets of generated and training samples in the material discovery domain,
which is limited in providing sub-feature or feature-level evaluation of
models. GuacaMol [18] is one of the early benchmark platforms for new molecule
discovery, which aims to evaluate generative models across different tasks,
e.g., fidelity and novelty. Molecular Sets (MOSES) [17] is another
benchmarking framework, which provides training and testing datasets, and a
set of metrics to evaluate the quality and diversity of generated structures
to standardize training and model comparisons.

Furthermore, automated characterization of subsets of samples, generated with
more or less frequency, i.e., generation frequency analysis, also still
remains challenging as the focus is more on latent- or feature-based
evaluation. Overall, the challenges associated with evaluating generative
models could be summarized as follows. First, multiple evaluation metrics are
model-dependent. For example, FCD [23] depends on latent representation, and
Maximum-mean discrepancy [24] is more specifically used to evaluate graph-
based generative models. State-of-the-art metrics also suffer from limited
generalizability (across different levels of feature interactions) and
interpretability, e.g., by domain experts, which is critical to achieve
trustworthy AI solutions [25]. In addition, existing evaluation metrics are
susceptible to potential flaws in predictive models used in goal-oriented or
constrained generation. Moreover, existing evaluation strategies lack a
generic and standalone evaluation metric that combines both distributional
metrics (e.g., uniqueness and diversity) and property-based metrics that score
single property.. The dependency on a single-constraint objective lacks a
principled approach to incorporate multiple target features. This becomes a
significant challenge when a single and inaccurate evaluation metric is used,
which oversimplifies real discovery problems and hence less practical.

##  3 Proposed: MPEGO Framework

The proposed MPEGO framework (see Fig. 1) aims to provide an effective and
multi-level characterization of generative models. The multi-level evaluation
of MPEGO (see Fig. 2) starts from sub-feature-based low-level evaluation and
their step-by-step aggregation to provide high-level evaluations. In this
section, we first, formulate the critical research questions MPEGO framework
is designed to address, followed by the details on its core components.

###  3.1 Problem Statement

Figure 1: Overview of the MPEGO framework

Let
$\mathcal{G}_{1},\mathcal{G}_{2},\cdots,\mathcal{G}_{k},\cdots,\mathcal{G}_{K}$
be datasets comprising samples generated from $K$ black-box generative models
($\Theta_{1},\Theta_{2},\cdots,\Theta_{k},\cdots,\Theta_{K}$) trained on a
dataset $\mathcal{T}$. Can we evaluate the generation capability of each
$\Theta_{i}$ in a scalable, easily interpretable, and multi-objective manner?
Specifically, we aim to address two questions.

  * Q1:

Given a set of features characterizing the samples, how do we quantify the
generation capability of each model compared to another model or the training
data, based on one or more of these features, i.e., at different levels of
abstractions?

  * Q2:

What are the characteristics of samples being generated with extreme
frequencies (least or most) by each of the generative models, compared with an
other model or the training data, i.e., generation frequency analysis?

To address Q1, we propose a Hierarchical Independence Evaluation (HIE) that
aims to quantify the performance of generative models at different levels of
feature interactions hierarchically, starting with a sub-feature level
evaluation (e.g., a specific range of a feature) to the global aggregation of
multiple features. To address Q2, we employ multi-dimensional subset scanning
(MDSS) [26] that aims to automatically identify and characterize over- and
under-generated subsets of samples.

###  3.2 Feature Extraction and Pre-processing

Given representative examples of generated and training samples, MPEGO starts
with the extraction of $M$ features from these samples,
$\mathcal{F}=\\{f_{1},f_{2},\cdots,f_{m},\cdots,f_{M}\\}$. The type of feature
values could be binary, continuous, or categorical, and a further pre-
processing could be applied in the follow up steps. For example,
discretization of continuous features is required for sub-feature-level
performance evaluation, and MPEGO provides different discretization types,
e.g., equal width, equal frequency or based on $k$-means.

Figure 2: Details of Multi-level evaluation component of the MPEGO framework
that comprises Hierarchical Independence Evaluation (i.e., SIS, FIS, SAFIS and
GAFIS) and Generation Frequency Analysis (GFA). $A(\cdots)$ represents
aggregation operation, and anomalous subset refers to the logical combinations
of features that characterize samples generated with extreme frequencies.

###  3.3 Hierarchical Independence Evaluation (HIE)

HIE follows a bottom-up approach (from a sub-feature to global aggregation
levels), as shown in Fig. 2, and it evaluates the performance generative
models at different levels of feature interactions. The lowest level of
evaluation in HIE is the sub-feature level independence score (SIS), which
aims to quantify the performance of generative models across different ranges
or unique values per feature, e.g., based on a a molecular weight range of
$200-300$ Daltons. The second layer in HIE represents feature-level
independence score (FIS), which quantifies the generation performance for each
feature. To this end, FIS could be computed via aggregation of SIS values,
thereby providing a weighting strategy for SIS scores per feature. FIS values
could also be computed directly, without aggregating SIS values, using a
different choice of objective measure is directly applied on the whole
feature.The proposed MPEGO framework is flexible to utilize different
objective measures (see Appendix D of the Supplementary Material for details).
Below we describe the computation of SIS and FIS values, using Yule’s Y
coefficient [27] as selected objective measure. Note that $\mathcal{G}_{k}$
vs. $\mathcal{T}$ refers to a case where the comparison is between the $kth$
generative model and the training set $\mathcal{T}$. On the other hand,
$\mathcal{G}_{k}$ vs. $\mathcal{G}_{j}$ refers to a case when the evaluation
between the $jth$ and $kth$ models, where $j\neq k$. However, we stick with
the $\mathcal{G}_{k}$ vs. $\mathcal{T}$ comparison below in order to ease
readability.

Let $f_{m}\in\mathcal{F}$ is a feature with $C_{m}$ unique values or ranges,
i.e., $f_{m}=\\{f_{m}^{u}\\}$, $u\in[1,2,\cdots,C_{m}]$. Note that $C_{m}$ is
the number of unique values for categorical features or the number of bins
after discretization of continuous features. SIS computation requires the
stratification of both the generated ($\mathcal{G}_{k}$) and training
($\mathcal{T}$) datasets per each unique value/range $f_{m}^{u}$ resulting
$\mathcal{G}_{km}^{u}$ and $\mathcal{T}_{m}^{u}$, respectively. The
complimentary subsets are then $\widetilde{\mathcal{G}_{km}^{u}}$ and
$\widetilde{\mathcal{T}_{m}^{u}}$, respectively. Note that
$\widetilde{\mathcal{G}_{km}^{u}}=\mathcal{G}_{km}^{u}|(f_{m}\neq
f_{m}^{u})=\mathcal{G}_{k}-\mathcal{G}_{km}^{u}$ and
$\widetilde{\mathcal{T}_{m}^{u}}=\mathcal{T}_{m}^{u}|(f_{m}\neq
f_{m}^{u})=\mathcal{T}-\mathcal{T}_{m}^{u}$. Accordingly, a $2\times 2$ pivot
table is generated for each $f_{m}^{u}$ as:

| $(f_{m}=f_{m}^{u})$ | $(f_{m}\neq f_{m}^{u})$
---|---|---
$\mathcal{G}_{km}$ | $\alpha$ | $\beta$
$\mathcal{T}_{m}$ | $\delta$ | $\gamma$

where $\alpha$ is the number of generated samples in ${\mathcal{G}_{km}^{u}}$
that are characterized by the feature value $f_{m}=f_{m}^{u}$, $\beta$ is the
number of generated samples in ${\widetilde{\mathcal{G}_{km}^{u}}}$ with
$f_{m}\neq f_{m}^{u}$. Similarly, $\delta$ and $\gamma$ are the numbers of
training samples that satisfy $f_{m}=f_{m}^{u}$ in $\mathcal{T}_{m}^{u}$,
respectively. Note that $\alpha+\beta$ are the total numbers of generated
samples, i.e., $|{\mathcal{G}_{k}}|$. Similarly, $\delta+\gamma$ is the number
of training samples, i.e., $|\mathcal{T}|$. Then Yule’s Y coefficient is
computed from the pivot table as $o_{km}^{u}\in[-1,1]$:

$o_{km}^{u}=\frac{\sqrt{P(\mathcal{G}_{km}^{u})P(\widetilde{\mathcal{T}_{m}^{u}})}-\sqrt{P(\widetilde{\mathcal{G}_{km}^{u}})P(\mathcal{T}_{m}^{u})}}{\sqrt{P(\mathcal{G}_{km}^{u})P(\widetilde{\mathcal{T}_{m}^{u}})}+\sqrt{P(\widetilde{\mathcal{G}_{km}^{u}})P(\mathcal{T}_{m}^{u})}}$
(1)
$o_{km}^{u}=\frac{\sqrt{\alpha\gamma}-\sqrt{\beta\delta}}{\sqrt{\alpha\gamma}+\sqrt{\beta\delta}}$
(2)

SIS is then computed from $o_{km}^{u}$ value as $I_{km}^{u}=1-|o_{km}^{u}|$,
where $I_{km}^{u}\in[0,1]$ and higher $I_{km}^{u}$ reflects higher
independence between $\mathcal{G}_{k}$ and $\mathcal{T}$, i.e., SIS $=1$
represents complete independence. Feature-level Independence Score (FIS),
provides feature-based evaluation, i.e., higher abstraction than SIS.
Depending on the objective measure, FIS could be computed as 1) via a weighted
aggregation of SIS values, i.e.,
$I_{km}=\sum_{u=1}^{C_{m}}\lambda_{km}^{u}I_{km}^{u}$ where
$\sum_{u=1}^{C_{m}}\lambda_{km}^{u}=1$ and each $\lambda_{km}^{u}$ weights the
SIS value of $f_{m}^{u}$, or 2) via straightforward computation, without using
SIS values, when the objective measure is directly applied on each feature
without discretization, e.g., using Wasserstein distance. The third layer of
HIE is Selective Aggregation of Feature-level Independence Score (SAFIS),
which aims to aggregate FIS values from $R<M$ selected features in
$\mathcal{F}$. For example, features including scaffolding, fingerprints,
aromaticity, and the number of rings could be selected to reflect the
structural details of molecules in material discovery domain. The last layer
of HIE is Global Aggregation of Feature-level Independence Score (GAFIS),
which is computed via a weighted aggregation of all the FIS values in
$\mathcal{F}$. Note that SAFIS and GAFIS are computed as
$\hat{I}=\sum_{r=1}^{R}\eta_{r}I_{kr}$, where $\sum_{r=1}^{R}\eta_{r}=1$, and
$R<M$ for SAFIS and $R=M$ for GAFIS computation.

###  3.4 Generation Frequency Analysis (GFA)

Generative models trained on the same dataset will hardly generate samples
with exact characteristics. Thus, there is a potential over- or under-
generation of samples with certain characteristics. To this end, we employ
automated stratification of samples using multi-dimensional subset scanning
(MDSS) [26, 28] to identify subset of samples generated with divergent
frequencies. Specifically, to identify samples generated with divergent rates
by model $\Theta_{k}$, compared to the training set $\mathcal{T}$, we first
merge the corresponding datasets as
$\mathcal{D}=\mathcal{G}_{k}\cup\mathcal{T}$, and an outcome label ($y$) is
generated, such that $y_{i}=1$ for a sample in $\mathcal{G}_{k}$ and $y_{i}=0$
for a sample in $\mathcal{T}$. If there are $N_{g}=|\mathcal{G}_{k}|$
generated and $N_{t}=|\mathcal{T}|$ training samples in $\mathcal{D}$, the
expectation of generated samples in $\mathcal{D}$ is
$e_{g}=\frac{N_{g}}{N_{g}+N_{t}}$. Thus, GFA aims to identify a group of
samples with extreme deviations in their generation rate compared to $e_{g}$.
The deviation between the expectation and observation is evaluated by
maximizing a Bernoulli likelihood ratio scoring statistic, $\Gamma(\cdot)$.
The null hypothesis assumes that the odds of the generated sample in any
subgroup $\mathcal{S}$ is similar to the expected, i.e.,
$H_{0}:odds(\mathcal{S})=\frac{e_{g}}{1-e_{g}}$; while the alternative
hypothesis assumes a constant multiplicative increase in the odds of the
generated samples in $\mathcal{S}$, i.e.,
$H_{1}:odds(\mathcal{S})=q\frac{e_{g}}{1-e_{g}}$ where $q\neq 1$. Note that
$q>1$ for over-generated subset, and $0<q<1$ for under-generated subset. The
divergence score for a subgroup ($\mathcal{S}$) with reference $\mathcal{D}$
is formulated as, $\Gamma(\mathcal{S},\mathcal{D})$ and computed as:

$\Gamma(\mathcal{S},\mathcal{D})=\max_{q}log(q)\sum_{i\in
S}y_{i}-N_{s}*log(1-e_{g}+qe_{g}),$ (3)

where $N_{s}$ is the number of samples in $\mathcal{S}$. The divergent subset,
$\mathcal{S}$, identification is iterated until convergence to a local maximum
is found, and the global maximum is subsequently optimized using multiple
random restarts.

##  4 Experimental Setup

###  4.1 Training Datasets

We employed three different datasets in material discovery domain to validate
our MPEGO framework.

#### ZINC-250K

We utilize the publicly available
ZINC-250K222https://www.kaggle.com/datasets/basu369victor/zinc250k dataset,
which contains $249,455$ small molecules in Simplified Molecular-Input Line-
Entry System (SMILES) representation. Details on ZINC tool are available in
[22].

#### MOSES

The benchmark platform MOSES [17], besides implementing popular molecular
generation models and metrics, MOSES contains a refined dataset from
ZINC333https://zinc.docking.org/. The dataset has approximately 2M molecules
in total, filtered by certain parameters such as molecular weight ranges, and
number of rotatable bonds, among others.

#### CIRCA

We used IBM’s Chemical Information Resources for Cognitive Analytics (CIRCA)
platform444https://circa.res.ibm.com/ to construct a dataset of organic salts
relevant to production of semiconductors via photolitography with chemical
amplification. Finally, we were able to evaluate environmental and
toxicological properties of 866 anions, that comprise the dataset used in this
study. Steps conducted to obtain final version of the dataset could be found
in Appendix A in the Supplementary Material.

Figure 3: Visualization of the CIRCA dataset and some of the structural
fingerprints relevant to MPEGO analysis (inset). The CIRCA dataset is
represented as a similarity network, where nodes correspond to anions and
links connect nodes if Dice similarity of the respective anions is at least
0.5. Node colors encode clustering recovered via modularity analysis and node
size encodes the number of connections of the node. Chemical structures are
included for the most connected nodes in the main clusters.

###  4.2 Generative Models

We utilized multiple generative models for our validation. Particularly, Graph
Convolutional Policy Network (GCPN) [19] and a Flow-based Autoregressive
(GraphAF) [20] were trained separately on ZINC-250K and MOSES datasets.
Subsequently, $10,000$ valid molecules were generated from each model. We rely
on GT4SD [29] for model implementations (experimental set-ups shown in
Appendix B). Similarly, from the MOSES-trained GCPN and GraphAF, $14665$ and
$1680$ molecules were generated respectively. For the CIRCA data, we employed
MolGX [21] and the Regression Transformer [15]) and generated 5000 new anions
with MolGX and 32617 new anions with Transformer. Note that balanced number of
samples were selected across datasets during our experimentation, i.e.,
$10000$, $1680$ and $5000$ samples were randomly selected for ZINC-250K-based,
MOSES-based, CIRCA-based evaluations.

###  4.3 Feature Extraction and Preprocessing

We have extracted the following six features from SMILES representations of
molecules in ZINK-250K and MOSES datasets and their corresponding generated
datasets from GCPN and GrpahAF. The following features are selected for the
evaluation: Aromaticity, ESOL, LogP, Weight, QED, SCScore. On the other hand,
we extracted fingerprints (see Fig. 3) that reflect structural details from
CIRCA dataset and MolGX and Transformer generated datasets. Full details of
features can be found in the Appendix C of the Supplementary Material. In
cases when discretization of continuous features is required to compute the
SIS values, we employ Yule’s Y coefficient as our default objective measure as
it satisfies multiple key requirements [30]. We also employ equal frequency
discretization type as it better handles outliers in the data, with five bins
for ZINC and with three bins for CIRCA based evaluations. While aggregating
features for SAFIS and GAFIS computations using normal averaging. Note that
the proposed approach is flexible to utilize different discretization types,
weighting strategies, and objective measures.

###  4.4 Evaluation Metrics

Our evaluation metrics include HIE’s SIS, FIS, SAFIS, and GAFIS values that
quantify the generation independence of models. The independence score is
obtained by normalizing the objective measure values to $[0,1]$. Note
independence score $=1.0$ represents complete independence. We also utilize
the histogram of features to provide a qualitative comparison. The
characterization of generation frequency analysis involves using the logical
combination of feature values to describe the identified subgroup, the size of
the subgroup $N_{s}$, the odds ratio between $\mathcal{S}$ and
$\widetilde{\mathcal{S}}=\mathcal{D}-\mathcal{S}$, $95\%$ Confidence Interval
(CI) and empirical $p$ value. We also reported the divergence score of the
identified group from the expectation in GFA and elapsed time to identify the
group. All the experiments are conducted on a desktop machine, 2.9 GHz Quad-
Core Intel Core i7 (processor), and 16 GB 2133 MHz LPDDR3 (memory).

##  5 Results and Discussion

###  5.1 Hierarchical Independence Evaluation

Table 1 provides extended FIS values across each of the features considered
for comparing GCPN and GraphAF models with the training ZINC-250K
($\mathcal{G}vs.\mathcal{T}$) and between each other - head-to-head
($\mathcal{G}vs.\mathcal{G}$). GCPN’s GAFIS value of $0.882$, demonstrating
competitive generation independence with GraphAF with GAFIS $=0.821$. This is
further shown with their head-to-head comparison, with Aromaticity ($FIS=1.0$)
and LogP ($FIS=0.906$) features. Divergent characteristics are also
demonstrated when the two models were evaluated based on QED, Weight and ESOL
features, achieving FIS of $0.853$, $0.809$, and $0.662$, respectively in
their head-to-head comparison. SAFIS values are shown as an example of
aggregation synthetic metrics, QED and LogP, where GCPN model achieves
superiority compared to GraphAF. GAFIS values in the bottom row are derived
from global aggregation of FIS values above. Overall, the results show the
flexibility of the MPEGO framework to evaluate models across different levels
of feature interactions and comparison baselines, i.e., training data or
another generated datasets.

Furthermore, Fig. 4 demonstrates the benefits of sub-feature level evaluation
(SIS) of GCPN and GraphAF models trained on ZINC-250K, using Molecular Weight
as a feature example. Five ranges of the molecular weight in Fig. 4 (a)
resulted from the discretization of the feature necessary for SIS evaluations.
Similarly to Table 1, the comparison is performed between generative models
and with the training dataset. Results demonstrate that GCPN and GraphAF
generated molecules with similar weight characteristics but distinctively
different at a few particular ranges, i.e., GCPN generated molecules similar
to ZINC-250K with range $[258.1,308.39)$ whereas GraphAF showed better
resemblance with ZINC-250K at ranges $[201.63,258.1)$ and $\geq 361.47$
Daltons. Divergent sub-level generation characteristics is also encoded by
lower $\mathcal{G}$ vs. $\mathcal{G}$ independence score at those ranges. The
histogram plot in Fig. 4 (b) qualitatively compliments the insights from SIS
values in Fig. 4 (a) , where GraphAF is shown to generate more molecules with
extreme weight values. Note that such sub-feature level insights in Fig. 4 are
unique to our proposed framework as they are currently limited in the state-
of-the-art of generative model evaluation.

The results of comparisons of GCPN and GraphAF trained on MOSES dataset (Table
2) and MolGX and RT trained on CIRCA dataset (Table 3) demonstrate MPEGO’s
capabilities further. Comparisons between Table 1 with Table 2 show while
MOSES is used as a training, GraphAF achieved more independence than when
ZINC-250K is used, compared GCPN. Between MolGX and the RT, trained on CIRCA,
molecules generated from the RT achieved a higher resemblance with CIRCA than
MolGX. In their head-to-head comparison, MolGX and RT generated mostly
divergent structural details, as it is demonstrated by very low FIS values for
the majority of the fingerprints in $\mathcal{G}$ vs. $\mathcal{G}$ column of
Table 3.

Table 1: HIE scores for GCPN and GraphAF models trained on ZINC-250K dataset

|  |  $\mathcal{G}$ vs. $\mathcal{T}$ |  $\mathcal{G}$ vs. $\mathcal{G}$
---|---|---|---
|  | GCPN vs. | GraphAF vs. | GCPN vs.
Level |  | ZINC | ZINC | GraphAF
| Aromaticity | 1.000 | 1.000 | 1.000
| ESOL | 0.912 | 0.856 | 0.853
| LogP | 0.951 | 0.942 | 0.906
FIS | Weight | 0.656 | 0.713 | 0.809
| QED | 0.946 | 0.666 | 0.662
| SCScore | 0.825 | 0.752 | 0.866
SAFIS | QED+LogP | 0.949 | 0.804 | 0.784
GAFIS |  | 0.882 | 0.821 | 0.850

|  |  $\mathcal{G}$ vs. $\mathcal{T}$ |  $\mathcal{G}$ vs. $\mathcal{G}$
---|---|---|---
|  | GCPN vs. | GraphAF vs. | GCPN vs.
Level | Weight Range | ZINC | ZINC | GraphAF
|  $<$201.63 | 0.406 | 0.322 | 0.859
| [201.63 , 258.1) | 0.631 | 0.840 | 0.778
SIS | [258.1 , 308.39) | 0.986 | 0.824 | 0.810
| [308.39 , 361.47) | 0.671 | 0.637 | 0.962
|  $\geq$361.47 | 0.586 | 0.939 | 0.638

(a)

(b)

Figure 4: (a) Example of Molecular Weight-based SIS values that provide sub-
feature level evaluation of GCPN and GraphAF models trained on ZINC-250K; (b)
histogram densities to provide qualitative visualization of the SIS values in
(a) Table 2: HIE Scores for GCPN and GraphAF models trained on MOSES dataset

|  |  $\mathcal{G}$ vs. $\mathcal{T}$ |  $\mathcal{G}$ vs. $\mathcal{G}$
---|---|---|---
|  | GCPN vs. | GraphAF vs. | GCPN vs.
Level |  | ZINC | ZINC | GraphAF
| Aromaticity | 1.000 | 1.000 | 1.000
| ESOL | 0.419 | 0.672 | 0.491
| LogP | 0.219 | 0.682 | 0.319
FIS | Weight | 0.550 | 0.491 | 0.760
| QED | 0.471 | 0.576 | 0.735
| SCScore | 0.846 | 0.865 | 0.744
SAFIS | QED+LogP | 0.345 | 0.629 | 0.527
GAFIS |  | 0.584 | 0.714 | 0.675

Table 3: HIE Scores for MolGX and Regression Transformer (RT) models trained
on CIRCA dataset

|  |  $\mathcal{G}$ vs. $\mathcal{T}$ |  $\mathcal{G}$ vs. $\mathcal{G}$
---|---|---|---
|  | MolGX vs. | Transformer vs. | MolGX vs.
Level | Fingerprint | MOSES | MOSES | Transformer
| 951226070 | 1.000 | 1.000 | 1.000
| 3218693969 | 0.599 | 0.876 | 0.500
| 2968968094 | 1.000 | 1.000 | 1.000
FIS | 882399112 | 1.000 | 1.000 | 1.000
| 2245384272 | 0.998 | 0.847 | 0.845
| 2246703798 | 1.000 | 1.000 | 1.000
| 3217380708 | 0.411 | 0.661 | 0.226
GAFIS |  | 0.858 | 0.912 | 0.796

###  5.2 Generation Frequency Analysis

Table 4 shows the divergence generation frequency of models, compared to the
training or other generated set. For example, the GCPN model generated
molecules with higher LogP ($\geq 1.31$) and lower weight ($<258.1$ Daltons)
compared to the training ZINC-250K. When we compare the two models head-to-
head, GCPN tends to generate molecules with higher QED ($\geq 0.64$) values
compared to GraphAF ($<0.5$). When MOSES data is used as validation, LogP
values become the differentiation factor as GraphAF tends to generate lower
values ($<3.91$) compared to GCPN ($\geq 3.91$). On CIRCA dataset, MolGX
generates samples with a higher occurrence of ’3217380708’ fingerprint
compared to the RT model and the training CIRCA. The size of the identified
group, along with the multiplicative factor (q) and the odds ratio values,
confirm the significance of the identified divergent generation in our
generation frequency analysis.

Table 4: Results from generation frequency analysis, validated across three
datasets: ZINC250K (with GCPN and GraphAF models), MOSES (with GCPN and
GraphAF models), and CIRCA (with MolGX and RT). We also compared the models
headtohead. The $q$ factor represents the multiplicative increase in the odds
of the outcome in the identified group compared to the null hypothesis.
Elapsed time is in seconds.

Model | Baseline | Expected |  Subset | Group size | Observed | q factor | Odds ratio | 95% CI | pvalue | Score | Elapsed
---|---|---|---|---|---|---|---|---|---|---|---
GCPN | ZINC | 0.5 |  LogP$\geq 1.31$ & Weight $<258.1$ | 5228 | 0.888 | 7.952 | 13.98 | (12.75, 15.33) | 0.0 | 1781.619 | 9.21
| GraphAF | 0.5 |  SCScore $<3.54$ & QED $\geq 0.64$ & Weight $<362.48$ | 8040 | 0.797 | 3.923 | 9.14 | (8.54, 9.77) | 0.0 | 1493.252 | 10.99
GraphAF | ZINC | 0.5 |  SCScore $\geq 2.62$ & QED $<0.64$ & Weight $<258.1$ | 4072 | 0.946 | 17.425 | 27.71 | (24.11, 31.85) | 0.0 | 1949.670 | 8.96
| GCPN | 0.5 |  QED $<0.5$ | 5135 | 0.878 | 7.164 | 12.22 | (11.17, 13.37) | 0.0 | 1647.798 | 7.73
GCPN | MOSES | 0.5 |  QED $<0.5$ & $2.36\leq$LogP$<3.91$ | 1422 | 0.985 | 66.714 | 397.89 | (253.9, 623.53) | 0.0 | 870.290 | 4.00
| GraphAF | 0.5 |  LogP $\geq 3.91$ & Weight $<298.32$ | 1284 | 0.992 | 127.400 | 525.01 | (279.16, 987.37) | 0.0 | 825.489 | 3.43
GraphAF | MOSES | 0.5 |  Weight $<250.38$ & $-4.75\leq$ ESOL$<-3.59$ | 915 | 0.991 | 113.375 | 245.40 | (121.72, 494.77) | 0.0 | 584.349 | 4.21
| GCPN | 0.5 |  LogP $<3.91$ & ESOL $<-4.75$ OR ESOL $\geq-3.59$ & Weight $<250.38$ | 1297 | 0.976 | 40.839 | 162.96 | (112.31, 236.46) | 0.0 | 744.637 | 3.33
RT | CIRCA | 0.5 |  2245384272 $\geq 1.0$ | 603 | 0.561 | 1.275 | 1.85 | (1.43, 2.39) | 0.0 | 2.430 | 1.92
MolGX | 0.5 |  3217380708 $<4.0$ & 3218693969 $<2.0$ | 353 | 0.989 | 87.250 | 286.60 | (105.19, 780.82) | 0.0 | 218.783 | 2.27
MolGX | CIRCA | 0.5 |  3217380708 $\geq 4.0$ | 484 | 0.804 | 4.095 | 14.94 | (10.99, 20.31) | 0.0 | 93.805 | 2.30
RT | 0.5 |  3217380708 $\geq 4.0$ | 416 | 0.935 | 14.407 | 61.39 | (39.48, 95.47) | 0.0 | 186.404 | 2.11

###  5.3 Ablation Study

We conducted ablation studies to validate the robustness of MPEGO for
different design choices of objective measures (see Fig. 5) and discretization
types (see Fig. 6). Though objective measures could vary in whether they
require discretization or not, the similar shapes of the plots in Fig. 5
demonstrate similar ranking performance across features. For example, in GCPN
vs. ZINC comparison (Fig. 5 (a)), Molecular weight achieved the least score
among features under all the different objective measures employed. The same
is true for QED in the GraphAF vs. ZINC validation in Fig. 5 (b). We also
validated the impact of the discretization types as shown in Fig.6 (a) and
(b). In both cases, _equal width_ and _$k$ -means_ discretizations provide
similar patterns while _equal frequency_ discretization demonstrated a
slightly different pattern, particularly for features with skewed distribution
as _A_ romaticity. Overall, all three types of discretization provide
competitive performance scores, thereby validating the stability of our MPEGO
framework. More ablation results are in Appendix E of the Supplementary
Material.

(a)

(b)

Figure 5: Different objective measures provided similar ranking per feature-
level performance score (FIS) of the generative models

(a)

(b)

Figure 6: Different discretization types provided similar ranking per feature-
level performance score (FIS) of the generative models

##  6 Conclusion and Future Work

We proposed MPEGO - a simple, generalizable, and model-agnostic evaluation
framework of generative models validated for material discovery domain. MPEGO
consists of two main performance evaluation blocks: Hierarchical Independence
Evaluation (HIE) and Generation Frequency Analysis (GFA). HIE follows a
bottom-up approach to quantify the generation performance of a model, starting
from per sub-feature level (at the bottom) to the global aggregation of
features (at the top). Thus, HIE provides a flexible performance evaluation of
generative models. Particularly by evaluating the generation independence of
models compared with the training data or other generative models using an
objective measure set by a user. GFA is applied to detect and characterize
divergent generation characteristics. Different from the existing evaluation
platforms, MPEGO provides interpretable insights that aim to facilitate
interactions with subject matter experts, which is crucial to develop
Trustworthy AI solutions. The proposed MPEGO toolkit was validated with
multiple datasets (ZINC-250K, MOSES, and CIRCA) and generative models trained
on these datasets, including GCPN, GraphAF, MolGX, and RT. Conditioned on the
training samples in these datasets, GCPN, GraphAF, and RT achieved higher
generation independence, compared to their counterparts, GraphAF, GCPN, and
MolGX models, respectively. Future work aims to evaluate generative models
from different domains to further validate the domain-agnostic nature of the
MPEGO framework. We also plan to utilize MPEGO to improve the efficiency of
latent-based analyses, such as creativity characterization [31] and out-of-
distribution detection [32]. MPEGO will be natively integrated into the GT4SD,
the Generative Toolkit for Scientific Discovery [29] and the source code and
experiments will be available at: https://github.com/GT4SD/mpego.

## References

  * [1] Albert P Bartók, Sandip De, Carl Poelking, Noam Bernstein, James R Kermode, Gábor Csányi, and Michele Ceriotti.  Machine learning unifies the modeling of materials and molecules.  Science advances, 3(12):e1701816, 2017.
  * [2] O Anatole von Lilienfeld and Kieron Burke.  Retrospective on a decade of machine learning for chemical discovery.  Nature communications, 11(1):1–4, 2020.
  * [3] Tiago Sousa, João Correia, Vítor Pereira, and Miguel Rocha.  Generative deep learning for targeted compound design.  Journal of Chemical Information and Modeling, 61(11):5343–5361, 2021.
  * [4] Samuel C Hoffman, Vijil Chenthamarakshan, Kahini Wadhawan, Pin-Yu Chen, and Payel Das.  Optimizing molecules using efficient queries from property evaluations.  Nature Machine Intelligence, 4(1):21–31, 2022.
  * [5] Matthew D Lloyd.  High-throughput screening for the discovery of enzyme inhibitors.  Journal of Medicinal Chemistry, 63(19):10742–10772, 2020.
  * [6] Pavel G Polishchuk, Timur I Madzhidov, and Alexandre Varnek.  Estimation of the size of drug-like chemical space based on gdb-17 data.  Journal of computer-aided molecular design, 27(8):675–679, 2013\.
  * [7] Joseph A DiMasi, Henry G Grabowski, and Ronald W Hansen.  Innovation in the pharmaceutical industry: new estimates of r&d costs.  Journal of health economics, 47:20–33, 2016.
  * [8] Vijil Chenthamarakshan, Payel Das, Samuel Hoffman, Hendrik Strobelt, Inkit Padhi, Kar Wai Lim, Benjamin Hoover, Matteo Manica, Jannis Born, Teodoro Laino, et al.  Cogmol: target-specific and selective drug design for covid-19 using deep generative models.  Advances in Neural Information Processing Systems, 33:4320–4332, 2020.
  * [9] Jannis Born, Matteo Manica, Joris Cadow, Greta Markert, Nil Adell Mill, Modestas Filipavicius, Nikita Janakarajan, Antonio Cardinale, Teodoro Laino, and María Rodríguez Martínez.  Data-driven molecular design for discovery and synthesis of novel ligands: a case study on SARS-CoV-2.  Machine Learning: Science and Technology, 2(2):025024, 2021.
  * [10] Jannis Born, Tien Huynh, Astrid Stroobants, Wendy D Cornell, and Matteo Manica.  Active site sequence representations of human kinases outperform full sequence representations for affinity prediction and inhibitor generation: 3d effects in a 1d model.  Journal of Chemical Information and Modeling, 62(2):240–257, 2021\.
  * [11] Oscar Méndez-Lucio, Benoit Baillif, Djork-Arné Clevert, David Rouquié, and Joerg Wichard.  De novo generation of hit-like molecules from gene expression signatures using artificial intelligence.  Nature communications, 11(1):1–10, 2020.
  * [12] Łukasz Maziarka, Agnieszka Pocha, Jan Kaczmarczyk, Krzysztof Rataj, Tomasz Danel, and Michał Warchoł.  Mol-cyclegan: a generative model for molecular optimization.  Journal of Cheminformatics, 12(1):1–18, 2020.
  * [13] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang.  Geodiff: A geometric diffusion model for molecular conformation generation.  In The Tenth International Conference on Learning Representations, ICLR, 2022.
  * [14] Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al.  Biological sequence design with gflownets.  In International Conference on Machine Learning, pages 9786–9801. PMLR, 2022.
  * [15] Jannis Born and Matteo Manica.  Regression transformer: Concurrent conditional generation and regression by blending numerical and textual tokens.  Nature Machine Intelligence, 2023.  In press.
  * [16] Connor W Coley, Natalie S Eyke, and Klavs F Jensen.  Autonomous discovery in the chemical sciences part ii: outlook.  Angewandte Chemie International Edition, 59(52):23414–23436, 2020\.
  * [17] Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, et al.  Molecular sets (moses): a benchmarking platform for molecular generation models.  Frontiers in pharmacology, 11:1931, 2020.
  * [18] Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher.  Guacamol: benchmarking models for de novo molecular design.  Journal of chemical information and modeling, 59(3):1096–1108, 2019\.
  * [19] Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec.  Graph convolutional policy network for goal-directed molecular graph generation.  Advances in neural information processing systems, 31, 2018.
  * [20] Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang.  Graphaf: a flow-based autoregressive model for molecular graph generation.  arXiv preprint arXiv:2001.09382, 2020.
  * [21] Seiji Takeda, Toshiyuki Hama, Hsiang-Han Hsu, Victoria A Piunova, Dmitry Zubarev, Daniel P Sanders, Jed W Pitera, Makoto Kogoh, Takumi Hongo, Yenwei Cheng, et al.  Molecular inverse-design platform for material industries.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2961–2969, 2020.
  * [22] John J Irwin, Teague Sterling, Michael M Mysinger, Erin S Bolstad, and Ryan G Coleman.  Zinc: a free tool to discover chemistry for biology.  Journal of chemical information and modeling, 52(7):1757–1768, 2012\.
  * [23] Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, and Gunter Klambauer.  Frechet chemnet distance: a metric for generative models for molecules in drug discovery.  Journal of chemical information and modeling, 58(9):1736–1741, 2018\.
  * [24] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola.  A kernel two-sample test.  The Journal of Machine Learning Research, 13(1):723–773, 2012.
  * [25] Weixin Liang, Girmaw Abebe Tadesse, Daniel Ho, L Fei-Fei, Matei Zaharia, Ce Zhang, and James Zou.  Advances, challenges and opportunities in creating data for trustworthy ai.  Nature Machine Intelligence, 4(8):669–677, 2022.
  * [26] Daniel B Neill, Edward McFowland III, and Huanian Zheng.  Fast subset scan for multivariate event detection.  Statistics in Medicine, 32(13):2185–2208, 2013.
  * [27] G Udny Yule.  On the methods of measuring association between two attributes.  Journal of the Royal Statistical Society, 75(6):579–652, 1912.
  * [28] Girmaw Abebe Tadesse, William Ogallo, Celia Cintas, and Skyler Speakman.  Model-free feature selection to facilitate automatic discovery of divergent subgroups in tabular data.  arXiv preprint arXiv:2203.04386, 2022.
  * [29] Matteo Manica, Joris Cadow, Dimitrios Christofidellis, Ashish Dave, Jannis Born, Dean Clarke, Yves Gaetan Nana Teukam, Samuel C Hoffman, Matthew Buchan, Vijil Chenthamarakshan, et al.  Gt4sd: Generative toolkit for scientific discovery.  arXiv preprint arXiv:2207.03928, 2022.
  * [30] Pang-Ning Tan, Vipin Kumar, and Jaideep Srivastava.  Selecting the right objective measure for association analysis.  Information Systems, 29(4):293–313, 2004.
  * [31] Celia Cintas, Payel Das, Brian Quanz, Girmaw Abebe Tadesse, Skyler Speakman, and Pin-Yu Chen.  Towards creativity characterization of generative models via group-based subset scanning.  arXiv preprint arXiv:2203.00523, 2022.
  * [32] Hannah Kim, Girmaw Abebe Tadesse, Celia Cintas, Skyler Speakman, and Kush Varshney.  Out-of-distribution detection in dermatology using input perturbation and subset scanning.  In 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pages 1–4, 2022.

## Appendix A: Steps to prepare CIRCA dataset

Initially, we carried out a patent search in CIRCA for ionic photoacid
generators (PAGs) that are sulfonium or iodonium salts. We extracted anionic
components of the sulfonium and iodonium salts, finding 493 unique anions. We
ran additional CIRCA search to enrich the dataset with anions relevant to
photolithography using the following query: “PAG” OR “photo-acid generator” OR
“photo initiator” OR “acid generating agent” OR “photocationic initiator” OR
“photocationic polymerization initiator” OR “cationic initiator” OR “cationic
type initiator” The additional search returned ionic compounds with 1398
anions, 189 of these 1398 anions were among the initial 493 anions. We
processed the set of collected anions, keeping only anions with charge -1 and
more than 1 carbon atom, reducing the dataset to 1086 anions. Out of 1086
anions, we were able to evaluate environmental and toxicological properties of
866 anions, that comprise the dataset used in this study

## Appendix B: Experimental Setup for GCPN and GraphAF training

Table 5: Summary of hyper-parameters set-up to train the two graph-based
models: GCPN and GraphAF using ZINC-250K dataset.

| Models
---|---
Setting | GCPN | GraphAF
Input Dimension | 18 | 9
Number of relation | 3 | 3
Batch normalization | False | True
Atom types | [6-9, 15-17, 35, 53] | [6-9, 15-17, 35, 53]
Hidden dimensions | [256, 256, 256, 256] | [ 256, 256, 256]

## Appendix C: Descriptions of features

Table 6: Features extracted from samples generated from GCPN and GraphAF
models trained on ZINC-250k and MOSES datasets.

Feature |  Description
---|---
QED |  Qualitative Estimate of Drug-likeness
ESOL |  Estimated SOLubility
SCScore |  Synthetic Complexity Score
SAS |  Synthetic Accessibility Score.
Scaffold |  A molecule is identical to its scaffold
Ring |  A molecule contains a ring structure
LargeRing |  A molecule contains a ring structure with more than $6$ atoms
Aromatic |  A molecule contains an aromatic ring
SteroCharacter |  SMILES string contains a stereochemistry string.
Stereocenter |  Whether a molecule contains a stereocenter,
Heterocycle |  A molecule has at least one ring with at least two different atoms
Lipinski |  A molecule adheres to the Lipinski’s rule of five.
HBondDonor |  A molecule has not more than 5 hydrogen bond donors
HBondAcceptor |  A molecule has not more than 10 hydrogen bond acceptors
MolecularWeight |  The molecular mass in Daltons
LogP |  Logarithmic partition coefficient

Multiple fingerprints, informing the presence of particular structural
elements in a given molecule, are extracted from CIRCA dataset and samples
generated from MolGX and RT models. The following fingerprints are selected
for evaluation in this study due to their higher variance across the datasets.

Fingerprint |  Molecular Subgraph
---|---
2245384272 |
951226070 |
3217380708 |
2246703798 |
882399112 |
3218693969 |
2968968094 |
Table 7: Fingerprints (and their molecular subgraph visualizations) selected
for evaluations in CIRCA datasets and its associated models: MolGX and RT.

## Appendix D: List of Objective measures

The MPEGO framework could handle different objective measures to quantify sub-
feature or feature-level-based association between generated and training
datasets or between two generated datasets. The list of objective measures
implemented in the MPEGO framework includes: Odds ratio, Yule’s Y
Coefficients, Yule’s Q coefficients, Mutual Information,J Measure, Gini Index,
Wasserstein Distance, Kstest, Support, Confidence, Laplace, Collective
Strength, Kappa, Conviction, Interest, Cosine, Piatetsky Shapiro, Certainty
Factor, Added Value, Jaccard and Kolmogorov–Smirnov.

## Appendix E: Ablation results

Similar ablation studies are applied for head-to-head comparison of models as
shown in Fig. 7, where the robustness of the MPEGO framework is demonstrated
across varieties of objective measures (Fig. 7 (a)) and discretization types
(Fig. 7 (b)).

(a)

(b)

Figure 7: Results from the ablation study on the variations of (a) objective
measures and (b) discretization types on the FIS values, validated with a
head-to-head comparison of GCPN and GraphAF models, trained on ZINC-250K.


# Reduce, Reuse, Recycle: Is Perturbed Data better than Other Language
augmentation for Low Resource Self-Supervised Speech Models

###### Abstract

Self-supervised representation learning (SSRL) has improved the performance on
downstream phoneme recognition versus supervised models. Training SSRL models
requires a large amount of pre-training data and this poses a challenge for
low resource languages. A common approach is transferring knowledge from other
languages. Instead, we propose to use audio augmentation to pre-train SSRL
models in a low resource condition and evaluate phoneme recognition as
downstream task. We performed a systematic comparison of augmentation
techniques, namely: pitch variation, noise addition, accented target-language
speech and other language speech. We found combined augmentations
(noise/pitch) was the best augmentation strategy outperforming accent and
language knowledge transfer. We compared the performance with various
quantities and types of pre-training data. We examined the scaling factor of
augmented data to achieve equivalent performance to models pre-trained with
target domain speech. Our findings suggest that for resource constrained
languages, in-domain synthetic augmentation can outperform knowledge transfer
from accented or other language speech.

Index Terms—  self-supervised learning, low resource pre-training, phoneme
recognition, audio augmentation, autoregressive predictive coding

##  1 Introduction

Self-supervised representation learning (SSRL) has gained popularity in speech
processing [1, 2, 3] thanks to its ability to learn general-purpose feature
representations by using large datasets without human labels [4]. These models
learn hierarchical representations of the data by solving auxiliary tasks
where the labels are obtained from the data itself. Tasks such as phoneme
recognition and speech recognition are solved by finetuning SSRL models using
small annotated datasets. Figure 1 shows a typical SSRL model and its
application to downstream phoneme recognition task. As we can see the model
learns representations from the raw audio. The learned representations are
then used in the downstream task and fine-tuned with small amount of labelled
data.

Fig. 1: Pre-training and fine-tuning of SSRL Model Table 1: Augmentation strategy on pre-training Experiments | Pre-training Data (hrs) | Aug. ratio
---|---|---
Clean Data | 25,50,75,100 | N/A
Clean25 + [African accented English Data] | 50, 75, 100 | 1, 2, 3
Clean25 + [Chinese speech Data] | 50, 75, 100 | 1, 2, 3
Clean25 + [Noise-aug] | 50, 75, 100 | 1, 2, 3
Clean25 + [Pitch-aug] | 50, 75, 100 | 1, 2, 3
Clean25 + [Noise-aug, Pitch-aug] | 50, 75, 100,175,325,425,525 | 1,2,3,4,6,12,16,20

Training SSRL models requires large amount of pre-training data [5]. State of
the art SSRL models use cross-lingual pre-training data, with the assumptions
that all languages share a common phoneme set and that having more examples of
pre-training data will help for low resource conditions [6, 7]. The XLS-R
model [6] is pre-trained with data from 128 languages. However, some languages
such as Irish, Oriya and Manx have contributed only a small fraction of the
pre-training data in [6]. The performance of XLS-R on these low resource
languages are negatively affected due to the high imbalance of languages data
in pre-training. Improving the performance of these low resource languages has
not been explored systematically with respect to pre-training SSRL models.

Data augmentation is widely used approach for artificially increasing training
data in order to improve the performance of models trained on small datasets.
This approach has been used for image classification [8], text classification
[9] and speech recognition [10]. The proposed approach in [10] has used speed
perturbation for increasing the amount of training data. In speed
perturbation, the speed of the original audio is increased/decreased to
generate synthetic audio which capture more variability in the underline
training data. This approach is also useful for low resource training
conditions or domain mismatch conditions. For example, Ko et al. [11] proposed
to use different noise augmentations (such as additive noise sources,
reverberations, speed perturbations) to the training data in order to train
noise robust speech recognition model.

Recently audio augmentation has been applied to pre-training data for building
robust pre-trained models [12]. In [13], the contrastive predictive coding
(CPC) model is pre-trained with augmented data. Kharitonov et al. [13] used
different augmentation effects such as additive noise, reverberation and pitch
modification on the training data to improve the performance of a CPC-based
pre-trained model on various downstream tasks. More noise-augmentation
techniques have been applied to the wav2vec 2.0 model pre-training data in
[14, 15, 16]. Recent work [17] used the wav2vec 2.0 model pre-trained and
evaluated in low resource training conditions. Our work is similar to [17]
where we target low resource training conditions depending upon the
availability of the pre-training data.

In this work we systematically test different augmentation strategies for
scenarios where the amount of pre-training data for training SSRL models are
limited. We simulate our experiment using English language as a proxy for low
resource languages as it allows us to compare to pre-training with larger
amounts of data but our approach can be applied to low resource languages. We
first added African accented English speech to the pre-training data to
explore the effect of same language but different accent to the target data.
Then we added Chinese language speech to see the benefits of cross-lingual
data. Finally, we evaluated synthetic data augmentation of the pre-training
data by using noise augmentation, pitch modification, and both techniques
combined.

We arbitrarily chose a simple and effective SSRL model from [1] called
autoregressive predictive coding (APC). There are other variants of APC such
as [18, 19, 20, 21]. By definition, the APC model is a generative approach
which predicts future frames based on the previous history of frames. In this
work, we have trained the APC model in a low resource training environment
(i.e. with a small quantity of unlabelled data). We considered the
availability of low resource training data conditions ranging from 25-hours to
100-hours of Librispeech data. Then we fine-tuned the pre-trained APC model
with 10-hours of Librispeech phoneme labelled data. The fine-tuned model is
evaluated on Librispeech test-clean.

##  2 Experimental Setup

###  2.1 Experiments

In this work, we experimented with different augmentation strategies in pre-
training of the SSRL model in order to increase the training data samples. We
added distinct accent speech and other language speech to the pre-training
data of SSRL model. This will enable us to examine the effect of knowledge
transfer in pre-training the SSRL model. These are examples of cross-lingual
transfer in pre-training data. Next, we added noise-augmentation to perturb
the training data so that the pre-trained model learns robust representations.
For this purpose, we mixed training data with additive noise samples in the
time domain with 5dB, 10dB and 15dB signal to noise (SNR) ratio. We applied
pitch-modification to the training data so that the model learns pitch-
invariant representations. Then we applied combined pitch-modification and
noise-augmentation effects to the pre-training data and pre-trained the SSRL
model. The resultant pre-trained model learns pitch-invariant and robust
representations. In all these experiments, we used 80-dimension mel-
spectrogram features as input to the pre-trained model and phonemes as the
target labels in downstream task. We pre-trained the APC model with the
following clean and augmented data.

Clean Data: We first partitioned Librispeech train-clean-100 data into 25-hrs,
50-hrs, 75-hrs, 100-hrs and then used these subsets as clean augmented pre-
training data.

Clean25 + [African accented English Data]: We mixed baseline Librispeech
25-hrs with African accented English speech data as pre-training augmented
data.

Clean25 + [Chinese speech Data]: We added real Chinese speech data with the
baseline Librispeech 25-hrs in pre-training data.

Clean25 + [Pitch-aug]: We first applied synthetic pitch-modification effects
to baseline Librispeech 25-hrs and then combined the baseline and pitch-
augmented in pre-training data.

Clean25 + [Noise-aug]: We added noise augmentations from freesound database to
baseline 25-hrs data and then combined baseline 25-hrs and noise-augmented in
pre-training data.

Clean25 + [Noise-aug, Pitch-aug]: We added pitch-modification and noise-
augmentation effects to the baseline Librispeech 25-hrs and combined them in
the pre-training data. We increased the pre-trained augmented data with more
augmentation factors.

The pre-trained models were fine-tuned with 10-hrs of data labelled with
phonemes for the downstream phoneme recognition task. The fine-tuned models
were then evaluated on the Librispeech test-clean dataset in downstream task.
Note that in all these experiments we kept the fine-tuning and evaluation data
fixed. We only experimented and applied different augmentation strategies to
the pre-training data. For simplicity, all these experiments are shown in
Table 1.

###  2.2 Datasets

We used the Librispeech dataset [22], African accented English speech
dataset111https://zindi.africa/competitions/intron-afrispeech-200-automatic-
speech-recognition-challenge, Common Voice Chinese speech
dataset222https://commonvoice.mozilla.org/en/datasets and MUSAN database [23]
for pre-training the SSRL models. Librispeech dataset is an American English
reading speech widely used in the literature for research benchmarks. In this
work, we used Librispeech train-clean-100 dataset for pre-training which is
then partitioned into 25-hrs, 50-hrs, 75-hrs and 100-hrs subsets for data
availability purposes. We extracted 75-hrs training data from African accent
English speech challenge corpus. This 75-hrs of data is then mixed with
baseline Librispeech 25-hrs in pre-training stage. Similarly, we extracted
75-hrs from Common Voice Chinese speech data and mixed with baseline
Librispeech 25-hrs for pre-training purposes. For noise-augmentation, we used
MUSAN datasbase [23]. MUSAN database contains different kind of noise sources
such as background music, freesound, sound-bible and overlap noise speech. In
this work, we extracted freesound and sound-bible noise sources only and added
to the available pre-training data for noise augmentation. We also added pitch
modification to the pre-training data using WavAugment [24]. Note all these
speech datasets are used for pre-training the SSRL models.

For fine-tuning, we extracted a fixed 10-hrs of speech from Librispeech train-
clean-360 to test a downstream phoneme recognition task. The pre-trained model
was then evaluated on a phoneme recognition task. Librispeech standard test-
clean dataset is used for evaluation. The fine-tuned 10-hrs and test-clean
dataset used phoneme labels from [25].

###  2.3 APC Architecture

The architecture of the APC model is a Long Short-term Memory (LSTM) structure
with skip connections [26]. LSTM network consists of 3 layers. Each layer size
is 512. The input to the APC model is a sequence of mel-spectrogram features
while the output is 512 size feature representations of these spectrogram. The
training criteria of the APC model is the 3-step prediction of the future
frames. Mean square error (MSE) loss is computed between prediction and ground
truth frames. The APC model is pre-trained for 100 epochs with learning rate
0.0001 and batch size 32. Adam optimizer is used for neural network
optimization.

In the downstream phoneme recognition task, feed-forward linear layers are
attached to the last layer of the APC model. The input dimension of the linear
layer is 512 while the output dimension is the total number of phonemes. Log
softmax is used to convert the output of the final layer to probabilities.
Finally, the cross-entropy loss is computed between probabilities and ground
truth phoneme classes. Remaining hyper-parameters are same as [1].

##  3 Results

###  3.1 Comparison of Augmentation Strategies

Figure 2 show the amount of available pre-training data and the augmentation
strategy (accent-aug, language-aug, noise-aug, pitch-aug, mix [pitch,
noise]-aug) and the phoneme evaluation performance on the standard Librispeech
test-clean dataset.

We first extracted 25 hrs from train-clean-100 as available pre-training data
and pre-trained the SSRL model. We consider the phoneme classification
accuracy for Librispeech 25-hrs as the baseline accuracy: 51.5%. Then we pre-
trained SSRL model with Librispeech 100-hrs clean and 100-hrs augmented data
for a fair comparison in figure 2. We added Librispeech 75-hrs, African
accented English speech 75-hrs, Chinese speech 75-hrs, noise-augmented 75-hrs,
synthetic pitch-modified 75-hrs and combined noise-pitch-augmented-75-hrs to
the baseline 25-hrs as augmented pre-trained data. We pre-trained the SSRL
model on these augmented data and reported the delta phoneme classification
accuracy as compared to baseline in figure 2. Figure 2 shows the augmentation
strategies we adopted (x-axis), and the the delta (improvement percentage) of
phoneme accuracy compared to the baseline 51.5%. Figure 2 shows that adding
augmentation to the pre-training data improves the performance in phoneme
classification. Adding African accented English speech data show the smallest
improvement in the phoneme classification. Pitch-augmentation performed better
than augmenting with accented or other language data. The noise-augmentation
also improves phoneme accuracy but less than the other strategies tested.
However, Figure 2 highlights that combining synthetic pitch-modification and
noise-augmentation performs better than the transfer strategies (i.e. accent
and language) or the single augmentation strategies.

Fig. 2: Performance of 100-hrs augmented pre-training data (25-clean-hrs +
75-aug-hrs) compared to the baseline 25-clean-hrs (baseline acc. 51.5%)

Fig. 3: Performance of augmented vs clean data in pre-training

###  3.2 Augmentation Size Analysis

A second analysis is conducted to understand the influence of the number of
augmentation hours. In Figure 3 we plot the phoneme classification (%) against
the total number of hours in pre-training data for each augmentation strategy.
We report the phoneme classification accuracy for baseline at 25-hrs. We first
added Librispeech 25-hrs to the baseline 25-hrs and pre-trained the model on
50-hrs combined data. We then added Librispeech 50-hrs to the baseline 25-hrs
and pre-trained the SSRL model on 75-hrs pre-training data. In the next
experiment, we added Librispeech 75-hrs to the baseline 25-hrs and pre-trained
the model on combined 100-hrs data. We repeated the same procedure with
augmentation data as well. Overall, we performed 4 experiments for individual
(combined) augmentations for a fair comparison. We compared the performance of
phoneme classification for all augmentations. Figure 3 shows that adding audio
augmentations improved the performance. This improvement in performance is
visible from the steep slopes of all augmentation approaches in Figure 3.
However, some augmentation slopes are steeper than others. e.g: the slope for
adding target Librispeech augmented data is higher than others (blue plot).
Other augmentations such as accent, language, noise and pitch-modification
also improves the performance. However the performance of combined pitch and
noise augmentation is better than individual augmentations and cross-lingual
transfer (accent, language). We noticed that the phoneme classification
accuracy can be further improved by increasing the augmentation factors for
our best augmentation strategy.

We extended our experiments by adding 150-hrs, 300-hrs, 400-hrs and 500-hrs
[noise, pitch] augmentations to the pre-training data and performed the pre-
training process as shown in Figure 3. Using 525-hrs mix of augmentation in
pre-training data matched the performance of SSRL model that was pre-trained
using data matching the target, i.e. more data from Librispeech train-
clean-100 as shown in red and blue plots in figure 3.

From our results in Figure 3, we observed that adding augmentation to the
baseline 25-hrs pre-training data improves the performance on downstream
phoneme classification. To better understand how much augmentation data in
pre-training improves the phoneme accuracy, the inset in Figure 3 shows a
graph for achieving a certain level accuracy using our augmentation approach.
The y-axis shows the delta phoneme accuracy (%) between the standard
Librispeech train-clean-100 and the baseline 25-hrs, while the x-axis shows
the additional augmentation multiplier data used in pre-training data. With
75-hrs (3-times more data) added to baseline (clean25), we observed delta
phoneme classification improvement 5.6%. In order to achieve that level
phoneme accuracy, we need to add 17-times more augmentation data to the pre-
training data. We also noted that adding 3-times synthetic augmentation
achieved a 3.3% improvement which is 60% of the improvement compared to pre-
training with the same amount of Librispeech data as shown in subplot Figure
3. For the same accuracy, we require 1.6$\times$ more augmented data than
using more pre-training data from Librispeech – an option that might not be
available in resource constrained scenarios.

###  3.3 Misclassification Analysis

From Figures 2 and 3, we observe that adding a mix of augmentation to pre-
training data improves the phoneme classification accuracy in downstream task.
However, the above figures do not show the impact of augmentation on the
misclassification of individual phonemes. In Figure 4, we show the impact of
augmentation on the phonemes that were most frequently misclassified.

The predicted phonemes are on the x-axis with the y-axis showing the ground
truth phonemes. For clarity of illustration we only present the three most
frequently misclassifications and hide both the least frequent
misclassifications and correct classifications. Figure 4(a) shows results for
the SSRL model pre-trained with Librispeech baseline 25-hrs. Phonemes /Y/,
/Z/, /ZH/ are the most misclassified phonemes. The misclassification of these
phonemes can be due to the fact that the APC model is not robust to the closer
phonemes (e.g., /S/ and /Z/) or due to noisy labels in the fine-tuned data.
The confusion matrix shows that data augmentation is able to reduce
misclassified phonemes (Figure 4(c)) similarly to adding clean data (Figure
4(b)). This further demonstrates the ability of data augmentation to mitigate
issues such as noisy labels and improve APC feature robustness in resource
constrained scenarios. A further analysis should be conducted to understand
whether some augmentation techniques are better at specific phoneme classes
which could suggest applying a class-conditional data augmentation strategy,
if phoneme labels are available in the pre-training data.

Fig. 4: Improvement of most frequent misclassified phonemes from (a), (b) and
(c); x-axis show the predicted phonemes, y-axis show the ground truth phonemes

##  4 Conclusions

This study proposes various augmentation strategies when the amount of pre-
training data is limited. Our systematic analysis demonstrates that the
following augmentations applied to the pre-training data lead to better
performance in the SSRL model for the downstream phoneme recognition task:
inclusion of African accented English speech data, Chinese speech data, noise
augmentation, and pitch modification. In particular, we found that combining
noise and pitch augmentation yields more promising results than incorporating
data from different languages or accents. We scaled the augmentation for the
strategy with the best results and achieved the performance equivalent to the
performance of model pre-trained with target speech data with 17$\times$
augmentation. This paper highlights the potential of artificial data
augmentation as an effective technique for resource-constrained SSRL pre-
training, proving more beneficial than equivalent transferring knowledge from
alternative language or accents to the low-resource target. In future, we will
consider adding synthetic speech generated from text-to-speech (TTS) systems
in the SSRL model pre-training process and extend our work to both phoneme and
speech recognition downstream tasks.

## References

  * [1] Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James R. Glass,  “An unsupervised autoregressive model for speech representation learning,”  in INTERSPEECH, 2019.
  * [2] Aaron van den Oord, Yazhe Li, and Oriol Vinyals,  “Representation learning with contrastive predictive coding,” 2018.
  * [3] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Trans. Audio, Speech and Lang. Proc., vol. 29, pp. 3451–3460, oct 2021.
  * [4] Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdel rahman Mohamed, and Hung yi Lee,  “Superb: Speech processing universal performance benchmark,”  in Interspeech, 2021.
  * [5] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  Advances in neural information processing systems, vol. 33, pp. 12449–12460, 2020.
  * [6] Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al.,  “Xls-r: Self-supervised cross-lingual speech representation learning at scale,”  arXiv preprint arXiv:2111.09296, 2021.
  * [7] Thibault Sellam, Ankur Bapna, Joshua Camp, Diana Mackinnon, Ankur P Parikh, and Jason Riesa,  “Squid: Measuring speech naturalness in many languages,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [8] Luis Perez and Jason Wang,  “The effectiveness of data augmentation in image classification using deep learning,”  arXiv preprint arXiv:1712.04621, 2017.
  * [9] Markus Bayer, Marc-André Kaufhold, and Christian Reuter,  “A survey on data augmentation for text classification,”  ACM Computing Surveys, vol. 55, no. 7, pp. 1–39, 2022.
  * [10] Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur,  “Audio augmentation for speech recognition,”  in Sixteenth annual conference of the international speech communication association, 2015.
  * [11] Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L. Seltzer, and Sanjeev Khudanpur,  “A study on data augmentation of reverberant speech for robust speech recognition,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, pp. 5220–5224.
  * [12] Asad Ullah, Alessandro Ragano, and Andrew Hines,  “Improving phoneme recognition with augmented autoregressive predictive coding,”  in 2023 34th Irish Signals and Systems Conference (ISSC), 2023, pp. 1–6.
  * [13] Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Pierre-Emmanuel Mazaré, Matthijs Douze, and Emmanuel Dupoux,  “Data augmenting contrastive learning of speech representations in the time domain,”  in 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 215–222.
  * [14] Heming Wang, Yao Qian, Xiaofei Wang, Yiming Wang, Chengyi Wang, Shujie Liu, Takuya Yoshioka, Jinyu Li, and DeLiang Wang,  “Improving noise robustness of contrastive speech representation learning with speech reconstruction,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6062–6066.
  * [15] Qiu-Shi Zhu, Jie Zhang, Zi-Qiang Zhang, Ming-Hui Wu, Xin Fang, and Li-Rong Dai,  “A noise-robust self-supervised pre-training model based speech representation learning for automatic speech recognition,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 3174–3178.
  * [16] Yiming Wang, Jinyu Li, Heming Wang, Yao Qian, Chengyi Wang, and Yu Wu,  “Wav2vec-switch: Contrastive learning from original-noisy speech pairs for robust speech recognition,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 7097–7101.
  * [17] Anuroop Sriram, Michael Auli, and Alexei Baevski,  “Wav2vec-aug: Improved self-supervised training with limited data,”  in Interspeech, 2022.
  * [18] Yu-An Chung and James R. Glass,  “Generative pre-training for speech with autoregressive predictive coding,”  IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3497–3501, 2019.
  * [19] Yu-An Chung, Hao Tang, and James R. Glass,  “Vector-quantized autoregressive predictive coding,”  ArXiv, vol. abs/2005.08392, 2020.
  * [20] Yu-An Chung and James Glass,  “Improved speech representations with multi-target autoregressive predictive coding,”  in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020, pp. 2353–2358, Association for Computational Linguistics.
  * [21] Gene-Ping Yang, Sung-Lin Yeh, Yu-An Chung, James R. Glass, and Hao Tang,  “Autoregressive predictive coding: A comprehensive study,”  IEEE J. Sel. Top. Signal Process., vol. 16, no. 6, pp. 1380–1390, 2022.
  * [22] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: An asr corpus based on public domain audio books,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210.
  * [23] David Snyder, Guoguo Chen, and Daniel Povey,  “MUSAN: A Music, Speech, and Noise Corpus,” 2015,  arXiv:1510.08484v1.
  * [24] Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Pierre-Emmanuel Mazaré, Matthijs Douze, and Emmanuel Dupoux,  “Data augmenting contrastive learning of speech representations in the time domain,”  arXiv preprint arXiv:2007.00991, 2020.
  * [25] Andy T. Liu, Shu wen Yang, Po-Han Chi, Po-Chun Hsu, and Hung yi Lee,  “Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,”  IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6419–6423, 2019.
  * [26] Sepp Hochreiter and Jürgen Schmidhuber,  “Long short-term memory,”  Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.