Let $CC=\left(\Omega_{0},\ldots\Omega_{R}\right)$ be a combinatorial complex
of dimension $R\geq 2$. The Hodge Laplacian spectrum of the combinatorial
complex $CC$, denoted $\text{Spec}_{\text{Hodge}}(CC)$, is defined by:

$\text{Spec}_{\text{Hodge}}(CC)=\left(\text{Spec}_{\text{Hodge},r}(CC)\right)_{2\leq
r\leq R}$, where for all $r\in\llbracket 2,R\rrbracket$,
$\text{Spec}_{\text{Hodge},r}(CC)=\text{Spec}\left(\mathcal{L}\left(\Omega_{r}\right)\right)$
where $\mathcal{L}$ is the Hodge Laplacian operator defined in Subsection 4.1
and Spec is the function that returns the vector of the eigenvalues of an
endomorphism or a squared matrix, with their order of multiplicity.

To compare two combinatorial complexes using the Hodge Laplacian spectrum, we
can employ a distance measure such as Maximum Mean Discrepancy (MMD) (see
Section 3.4) to compare the distributions of eigenvalues. This comparison can
accommodate complexes of different dimensions by zero-padding the higher
dimensions of the lower-dimensional complexes.

Finally, we introduce the rank-$r$ metric. It assesses the distribution of
features or sizes of rank-$r$ cells within a combinatorial complex. This
metric allows us to compare the sizes and properties of generated complexes
with the original ones. The definition of the metric depends on whether
feature information is attached to rank-$r$ cells as follows:

  * •

If $f_{r}=1$, the metric computes the distribution of sizes of the rank-$r$
cells (how many nodes per cell).

  * •

If $f_{r}\neq 1$, the metric computes the distribution of features attached to
rank-$r$ cells.

Mathematically, we can write the following definition:

######  Definition 38 (Rank-$r$ metric).

Let $CC=\left(\Omega_{0},\ldots\Omega_{R}\right)$ be a combinatorial complex
of dimension $R\geq 2$. The Rank-$r$ metric of the combinatorial complex $CC$,
denoted $\beta_{r}(CC)$, is defined by:

$\bullet$ If $f_{r}=1$,
$\beta_{r}(CC)=\left(\beta_{r,i}\right)_{d_{min}^{r}\leq i\leq d_{max}^{r}}$,
where $\forall i\in\llbracket d_{min}^{r},d_{max}^{r}\rrbracket$,
$\beta_{r,i}=\sum_{j=1}^{|\mathcal{X}_{r}|}\mathbb{1}_{\\{|x_{j}^{r}|=i\\}}$.

The parameter $\beta_{r,i}$ represents the number of rank-$r$ cells of size
$i$. $|\cdot|$ is the operator that returns the cardinal (number of nodes) of
a cell.

$\bullet$ If $f_{r}\neq 1$, $\beta_{r}(CC)=\left(\beta_{r,k}\right)_{1\leq
k\leq f_{r}}$, where $\forall k\in\llbracket 1,f_{r}\rrbracket$,
$\beta_{r,k}=\sum_{j=1}^{|\mathcal{X}_{r}|}\mathbb{1}_{\\{\Omega_{r}[:,j,k]\neq
0\\}}$.

The parameter $\beta_{r,k}$ represents the number of rank-$r$ cells with a
feature equal to $k$.

In both cases, the metric provides valuable insights into the structural
characteristics of the combinatorial complexes, enabling comparisons between
generated and original distributions.

The theoretical aspect of this thesis being presented, we can now delve into
our implementation.

##  Chapter 5 Implementation

All the datasets employed in our experiments (Section 5.1) can be easily and
naturally elevated to combinatorial complexes of dimension 2 and can be
efficiently represented numerically using the representation (Theorem 2 in
Subsection 4.1). Henceforth, throughout the remainder of this thesis, we will
exclusively focus on combinatorial complexes of dimension 2.

Our models, layers, preprocessing scripts, metrics, plots, and diffusion
framework have been meticulously implemented using the Python programming
language [86]. The orbit metric [17] was developed in C++ [87] for optimal
performance. In the subsequent section, Section 5.1, we offer an in-depth
account of our experiments. Following that, in Section 5.2, we present the
software components developed for this thesis.

###  5.1 Experiments

For our experiments, we carefully selected the datasets to have varying sizes
and characteristics, for example, synthetic graphs, real-world graphs, social
graphs or biochemical graphs, and to be able to compare the results with
different existing models and approaches for the sub-problem that is graph
generation. We also made sure that these datasets have been evaluated on other
methods to benchmark our framework.

We compared our proposed method against several general graph generative
models, each employing different architectural approaches. DeepGMG [88] and
GraphRNN [22] employ RNN-based architectures while GraphAF [67], GraphDF [68]
utilize flow-based architectures, and GRAPHARM [50] adopts diffusion based
architecture. These models are all autoregressive, generating graphs step by
step. On the other hand, GraphVAE [24], GraphEBM [89], GDSS [17], EDP-GNN [52]
and SGGM+SLD [51] utilize VAE, and EBM, and score-based models respectively.
GNF [90] and MoFlow [66] employ a flow-based model. The above models, as well
as our framework CCSD, are all one-shot, generating the entire graph in one
step.

We adopted three types of SDEs, VESDE, VPSDE, and sub-VP SDE, as introduced by
Song et al. [32], for the diffusion processes of each component. Additionally,
we employed either the PC sampler or the S4 solver to solve the system of
SDEs. Further implementation details for each dataset and the associated SDEs
can be found in Table 5.2 and Table 5.3. Similarly to GDSS, we created a
script to compute the Frobenius norm of the Jacobian of our models to assess
the complexity of learning partial scores, especially with respect to the
higher-order matrices.

As the training can take several days for a single dataset, we only trained
our models one time and sampled from them once. We acknowledge that this is a
limitation of our work but it could be easily fixed with more time. For the
GDSS baseline, we used the hyperparameters given by the original work.

####  5.1.1 Molecule Generation

For our molecule generation task, we utilized the QM9 dataset [91], which
comprises 133,885 molecules, each with a varying atom count ranging from 1 to
9. These molecules consist of Carbon (C), Fluorine (F), Oxygen (O), and
Nitrogen (N) atoms, with implicit Hydrogen atoms. The bonds in these molecules
can be single, double, or triple. 10% of the dataset is set apart for testing
purposes.

To prepare the molecules for processing, we converted each one into a graph
representation. These graphs have node features denoted as
$X\in\\{0,1\\}^{N\times f_{0}}$ and an adjacency matrix represented as
$A\in{0,1,2,3}^{N\texttimes N}$. Here, $N$ signifies the maximum number of
atoms found in any molecule in the QM9 dataset (which is 9 for QM9), and
$f_{0}$ represents the number of possible bond types/entries for the adjacency
matrix (4 for QM9). The entries in the adjacency matrix $A$ signify the types
of bonds between atoms, such as single, double, or triple bonds. We lifted the
graphs into CCs by transforming the ring into rank-2 cells (see Figure 3.2).

We followed a standard preprocessing procedure [67, 68] for these molecules.
This involved kekulization of the molecules using the RDKit Python library
[92] and removing hydrogen atoms. We also applied a valency correction method
proposed by Zang and Wang [66].

For our approach, we utilized the signal-to-noise ratio (SNR) and scale
coefficient obtained through a grid search in GDSS [17] as the basis for our
molecule generation. The optimization process in GDSS aimed to find the best
FCD (Fréchet ChemNet Distance) value among those that achieved a novelty score
exceeding 85%. This procedure has been chosen by the authors as a low novelty
value could potentially lead to low FCD and NSPDK MMD values.

After generating samples using the reverse diffusion process, we quantized the
adjacency matrices to values in the set $\\{0,1,2,3\\}$ by mapping them as
follows:

  * •

Values in the range $]-\infty,0.5]$ were mapped to $0$.

  * •

Values in the range $[0.5,1.5[$ were mapped to $1$.

  * •

Values in the range $[1.5,2.5[$ were mapped to $2$.

  * •

Values in the range $[2.5,+\infty[$ were mapped to $3$.

For evaluation, we used the MMD distance to compare the distributions of the
NSPDK statistic between the same number of generated and test molecules. We
compared also the molecular metrics introduced in Section 3.4, such as
validity, novelty, FCD, etc.

####  5.1.2 Graph Generation

We evaluated the performance of CCSD by examining the quality of generated
samples on a diverse set of graph datasets, encompassing both synthetic and
real-world graphs of varying sizes. Our evaluation datasets include:

  * •

Ego-small: A collection of 200 small ego graphs, extracted from the larger
Citeseer network dataset [93].

  * •

Community-small: A set of 100 randomly generated community graphs, following
the methodology introduced by Niu et al. [52].

  * •

Enzymes small: Comprising 35 graphs extracted from a pool of 587 protein
graphs (original Enzymes dataset), where we selected the graphs with fewer
than 12 nodes. These protein graphs represent the tertiary structures of
enzymes from the BRENDA database [94]

  * •

Grid small: A collection of 100 standard 2D grid graphs with varying
dimensions, ranging from 4x4 to 7x7 rows and columns (equivalent to 10x10 to
19x19 rows and columns in the original grid dataset).

The grid dataset generation and graph manipulation were carried out using the
NetworkX Python library [95]. We lifted the graphs into CCs by either applying
a loop-based lift procedure like for the molecules or a path-based lift
procedure (see Figure 3.3). More information about which lifting procedure is
applied to which dataset can be found in the parameter table (Table 5.2).

To ensure a fair comparison, we followed the experimental and evaluation
settings outlined by You et al. [22], including the same train/test split. We
employed the Maximum Mean Discrepancy (MMD) distance to compare the
distributions of various graph statistics between the generated samples and
the test graphs. The statistics we analyzed include degree distributions,
clustering coefficients, and the occurrences of 4-node orbits (to capture
higher-level motifs) [96]. We provided detailed definitions of these metrics
in Section 3.4.

In line with previous work [17], we used the Gaussian Earth Mover’s Distance
(EMD) kernel for computing MMDs, instead of the total variation (TV) distance
employed in some other papers, like Liao et al. [97]. This choice is made to
avoid an indefinite kernel and undefined behaviour [98].

For a fair evaluation of the generic graph generation task, we adhered to the
standard settings established by existing works [22, 90, 52], which cover
everything from node features to data splitting. Specifically, for CCSD, we
initialized the node features using one-hot encoding based on the degrees of
the nodes.

Our approach to graph generation also leverages the signal-to-noise ratio
(SNR) and scale coefficient determined through an extensive grid search
conducted in GDSS [17]. The optimization process that they used aims to find
the optimal MMD value while minimizing the average of three key graph
statistics: degree, clustering coefficient, and orbit. Additionally, we also
incorporated an exponential moving average (EMA) [99] for larger graph
datasets, such as Enzymes small and Grid small, to reduce the variance and
enhance performance.

After generating the samples by simulating the reverse diffusion process, we
quantized the entries of the adjacency matrices using the
$\mathbb{1}_{\\{x>0.5\\}}$ operator to obtain an adjacency matrix with values
in $\\{0,1\\}$.

####  5.1.3 Tanimoto similarity

The Tanimoto similarity metric plays a crucial role in molecule generation
tasks, offering a quantitative means to gauge the structural likeness between
newly generated molecules and those present in the training dataset [100].
This metric relies on the Morgan fingerprints [101], a widely accepted
molecular representation method. By leveraging these fingerprints’ similarity,
the Tanimoto similarity provides an effective way to assess how well the
generated molecules align with the structural attributes of the training
molecules. A higher Tanimoto similarity score signifies a closer structural
resemblance, showcasing the model’s ability to produce molecules that closely
mirror the characteristics of the original dataset. The Tanimoto Similarity
($\mathcal{T}$) can be mathematically expressed as:

$\mathcal{T}(A,B)=\frac{|A\cap B|}{|A\cup B|}$,

where $A$ and $B$ are sets of Morgan fingerprints. The resulting similarity
score ranges from 0 (indicating complete dissimilarity) to 1 (representing
perfect similarity, where the sets of features are identical).

####  5.1.4 Ablation Study & Models

We conducted experiments employing two different approaches. Firstly, for the
partial score function with respect to the adjacency matrix, denoted as
$s_{\theta_{1}}(CC_{t})\approx\nabla_{\Omega_{1,t}}\log\left(p_{t}\left(CC_{t}\right)\right)$,
we utilized the ScoreNetworkA_CC model, which incorporates an attention system
tailored for our specific generation task. This approach is called CCSD.
Conversely, we trained an alternative model called CCSD Base, which employs
the ScoreNetworkA_Base_CC model, consisting solely of MLP layers. These model
descriptions were detailed in Section 4.3.

Our ablation study aims to compare the attention-based model with the vanilla
variant that relies solely on MLP layers. Furthermore, we evaluated the
results of these two models alongside GDSS [17], the graph generation
framework based on stochastic differential equations that served as the
foundation for our framework. To ensure a fair comparison, we fine-tuned the
parameters of both CCSD and CCSD Base models to make them relatively similar
to those used in GDSS. We also ensured that the total number of trainable
parameters was roughly comparable, as illustrated in Table 5.1. Additionally,
we limited the depth of the MLP layers to minimize disparities in terms of
inference speed.

####  5.1.5 Parameters

In Table 5.1, we provided an overview of the number of parameters in each of
the score networks for every dataset. The trainable parameters represent the
cumulative count across all three score networks. Additionally, we reported
the maximum percentage difference between the models with the fewest and most
trainable parameters to assess that they do not differ by an order of
magnitude.

| GDSS | CCSD | CCSD Base | MAP (%)$\downarrow$
---|---|---|---|---
Ego-small | 179806 | 217633 | 144470 | 33.62
Community-small | 206803 | 227023 | 222838 | 8.91
Enzymes-small | 386179 | 390410 | 375160 | 3.91
Grid-small | 373234 | 436536 | 280839 | 35.67
QM9 | 32389 | 42764 | 34626 | 24.26
Table 5.1: Number of parameters for each dataset. We compared the number of
parameters between GDSS, CCSD, CCSD Base, and calculated the difference in
percentage between the models with the fewest and most trainable parameters.

In Table 5.2 and Table 5.3, we present the hyperparameters of the two
approaches for all the datasets.

| Hyperparameters | Ego-small | Community-small | Enzymes-small | Grid-small | QM9
---|---|---|---|---|---|---
General | Batch Normalization | False | False | False | False | False
Block layer | GCN | GCN | GCN | GCN | GCN
| Node init method | deg | deg | deg | deg | atom
| Lift procedure | cycles | path_based | cycles | path_based | cycles
| Lift kwargs |  | 3 | - | 3 | -
| $d_{min}$ | 3 | 3 | 3 | 3 | 3
| $d_{max}$ | 5 | 3 | 4 | 3 | 9
| Max node number | 18 | 20 | 12 | 49 | 9
| Min node value | 1 | 1 | 1 | 1 | 6
| Max node value | 1 | 1 | 1 | 1 | 9
| Max feature number | 17 | 10 | 10 | 5 | 4
| Min edge value | 1 | 1 | 1 | 1 | 1
| Max edge value | 1 | 1 | 1 | 1 | 3
$s_{\theta_{0}}$ | Number of GCN layers | 2 | 3 | 5 | 4 | 2
Hidden dimension | 12 | 32 | 32 | 24 | 10
$s_{\theta_{1}}$ | Number of attention heads | 4 | 4 | 4 | 4 | 4
Number of initial channels | 2 | 2 | 2 | 2 | 2
Number of hidden channels | 5 | 8 | 8 | 6 | 8
Number of final channels | 4 | 4 | 4 | 4 | 4
Number of Block layers | 2 | 5 | 6 | 6 | 3
Number layers in MLP | 2 | 2 | 2 | 2 | 3
| Hidden dimension | 12 | 32 | 32 | 24 | 10
| Hodge Block layer | HCN | HCN | HCN | HCN | HCN
| Number of Hodge Block layers | 1 | 1 | 2 | 1 | 2
| Number layers in Hodge MLP | 1 | 1 | 2 | 1 | 1
| Hidden dimension in Hodge MLP | 4 | 4 | 4 | 4 | 4
| Number of hidden Hodge channels | 2 | 2 | 4 | 2 | 4
| Number of final Hodge channels | 2 | 2 | 2 | 2 | 2
| Number of Hodge attention head | 2 | 2 | 2 | 2 | 2
| Hodge attention dim | 4 | 4 | 4 | 4 | 4
$s_{\theta_{2}}$ | Power higher-order | 2 | 2 | 2 | 2 | 2
Number layers in MLP | 1 | 1 | 1 | 1 | 1
Apply Hodge masks | True | True | True | True | True
SDE for $X$ | Type | VP | VP | VP | VP | VE
Number of sampling steps | 1000 | 1000 | 1000 | 1000 | 1000
$\beta_{min}$ | 0.1 | 0.1 | 0.1 | 0.1 | 0.1
$\beta_{max}$ | 1 | 1 | 1 | 1 | 1
SDE for $A$ | Type | VP | VP | VE | VP | VE
Number of sampling steps | 1000 | 1000 | 1000 | 1000 | 1000
$\beta_{min}$ | 0.1 | 0.1 | 0.2 | 0.2 | 0.1
$\beta_{max}$ | 1 | 1 | 1 | 0.8 | 1
SDE for $F$ | Type | VP | VP | VE | VP | VE
Number of sampling steps | 1000 | 1000 | 1000 | 1000 | 1000
$\beta_{min}$ | 0.1 | 0.1 | 0.1 | 0.1 | 0.1
$\beta_{max}$ | 1 | 1 | 1 | 1 | 1
Solver | Type | EM | EM + Langevin | S4 | Rev. + Langevin | Rev. + Langevin
SNR | - | 0.05 | 0.15 | 0.1 | 0.2
Scale coefficient | - | 0.7 | 0.7 | 0.7 | 0.7
Train | Optimizer | Adam | Adam | Adam | Adam | Adam
Learning rate | $1\times 10^{-2}$ |  1 $\times 10^{-2}$ | $1\times 10^{-2}$ | $1\times 10^{-2}$ | $5\times 10^{-3}$
Weight decay | $1\times 10^{-4}$ | $1\times 10^{-4}$ | $1\times 10^{-4}$ | $1\times 10^{-4}$ | $1\times 10^{-4}$
Batch size | 128 | 128 | 64 | 8 | 1024
Number of epochs | 5000 | 5000 | 5000 | 5000 | 300
EMA | - | - | 0.999 | 0.999 | -
Table 5.2: Hyperparameters of CCSD used in the generic graph generation tasks and the molecule generation tasks. We provide the hyperparameters of the score-based models ($s_{\theta_{0}}=\text{ScoreNetworkX}$, $s_{\theta_{1}}=\text{ScoreNetworkA\\_CC}$, and $s_{\theta_{2}}=\text{ScoreNetworkF}$), the diffusion processes (SDE for $X=\Omega_{0}$, $A=\Omega_{1}$, and $F=\Omega_{2}$), the SDE solver, and the training. $d_{min}$ and $d_{max}$ refer to the constrained on the rank-2 matrices generated via $\Omega_{2}=F$. | Hyperparameters | Ego-small | Community-small | Enzymes-small | Grid-small | QM9
---|---|---|---|---|---|---
$s_{\theta_{0}}$ | Number of GCN layers | 2 | 3 | 5 | 4 | 2
Hidden dimension | 32 | 32 | 32 | 24 | 10
$s_{\theta_{1}}$ | Number of attention heads | 4 | 4 | 4 | 4 | 4
Number of initial channels | 2 | 2 | 2 | 2 | 4
Number of hidden channels | 6 | 8 | 8 | 6 | 8
Number of final channels | 4 | 4 | 4 | 4 | 4
Number of Block layers | 4 | 5 | 6 | 6 | 3
Number layers in MLP | 2 | 2 | 2 | 2 | 3
| Hidden dimension | 32 | 32 | 32 | 24 | 10
| Hodge Block layer | HCN | HCN | HCN | HCN | HCN
| Number of Hodge Block layers | 3 | 2 | 2 | 2 | 2
| Number layers in Hodge MLP | 2 | 2 | 2 | 1 | 2
| Hidden dimension in Hodge MLP | 4 | 4 | 8 | 4 | 8
| Number of hidden Hodge channels | 4 | 4 | 8 | 2 | 8
| Number of final Hodge channels | 6 | 4 | 8 | 4 | 6
| Hidden dimension in Hodge layers | 6 | 4 | 8 | 2 | 8
$s_{\theta_{2}}$ | Power higher-order | 2 | 2 | 2 | 1 | 2
Number layers in MLP | 1 | 2 | 2 | 1 | 1
Apply Hodge masks | True | True | True | True | True
Table 5.3: Hyperparameters of CCSD Baseline used in the generic graph
generation tasks and the molecule generation tasks. This time, the score-based
model for the rank-2 cells is
$s_{\theta_{1}}=\text{ScoreNetworkA\\_Base\\_CC}$. The general parameters, the
SDEs, the solver and the hyperparameters for training are not displayed as
they are the same as CCSD.

###  5.2 Introducing CCSD: A Python library

In addition to the research outcomes presented above, we also delivered a
publicly available Python library, CCSD, designed to facilitate the
replication of our research results and to allow researchers to extend our
work conducted in this thesis. This comprehensive tool empowers users to train
and sample combinatorial complexes using the CCSD approach or generate graphs
and molecules employing the GDSS approach. The library offers extensive
functionality, including the creation of detailed logs and model checkpoints
during training and the generation of plots, animations, and object pickle
files during sampling. The library is technical and yet easy to use.

For easy access and utilization, CCSD is available on PyPi at the following
link: https://pypi.org/project/ccsd/. Additionally, its source code and
further information can be found on GitHub: https://github.com/AdrienC21/CCSD.

Figure 5.1: CCSD Logo.
Figure 5.2: PyPi page of the library.

To ensure the highest quality, our library has undergone meticulous linting,
thorough documentation, and extensive unit test coverage. We have strived to
offer a user-friendly interface, supported by comprehensive documentation (see
Subsection 5.2.1 below), and implemented a well-documented parser with a
detailed log system (Figure 5.3).

All researchers are more than welcome to contribute, whether it is through
debugging, adding functionnalities or new score network models.

Figure 5.3: Logs at the start of a sampling procedure.

####  5.2.1 Documentation

For complete information on CCSD, including installation instructions and
usage details, please refer to our comprehensive documentation available at:
https://ccsd.readthedocs.io/ (Figure 5.4). This documentation provides in-
depth insights into the package, including functions, parameters, input and
output types, and customization options. It is designed to assist users in
quickly grasping the concepts and making the most of our framework. Further
information can also be accessed on our GitHub page; repository:
AdrienC21/CCSD (Figure 5.5).

Figure 5.4: CCSD online documentation.
Figure 5.5: GitHub page of the library.

####  5.2.2 Example 1: Generating molecules by sampling combinatorial
complexes

In this example, we demonstrate how to use the CCSD package to sample
molecules from the learned QM9 dataset distribution. First, we import the
package, define the experiment parameters, and then sample 10,000 molecules.
The logs and SMILES representations of the generated molecules will be saved
in a folder named logs_sample, while visual materials such as combinatorial
complexes, graphs, molecules, and diffusion animations will be stored in a
sample folder.

Using a terminal, you can use the following bash command after cloning the
repository (recommended):

⬇

python main.py

–type sample

–config sample_qm9_CC

–folder ”./”

–comment ”Sample␣molecule␣through␣combinatorial␣complexes”

–seed 42

Or, after installing the library using the command pip install ccsd, you can
run the experiment using Python:

⬇

from ccsd.diffusion import CCSD

params = {

”type”: ”sample”,

”config”: ”sample_qm9_CC”,

”folder”: ”./”, # assume we are at the source of our project

”comment”: ”Sample␣molecule␣through␣combinatorial␣complexes”,

”seed”: 42 # optional

}

diffusion_model = CCSD(**params) # define the object

diffusion_model.run() # run the experiment

####  5.2.3 Example 2: Creating a combinatorial complex dataset

In this example, we generate a combinatorial complex dataset based on the
Community Small graph dataset.

After cloning the repository, run the following command:

⬇

python ccsd/data/data_generators.py –dataset community_small –is_cc

##  Chapter 6 Experiments Evaluation & Results

This chapter provides insights into the computational resources employed for
our experiments (Section 6.1), a quantitative evaluation of our models across
various datasets using the previously defined metrics (Section 6.2), and
visualizations of the results as part of a more qualitative evaluation
(Section 6.3).

###  6.1 Computing Resources

We developed this thesis using PyTorch [102] and trained our score network
models on a Nvidia RTX 2080 Ti, Nvidia Tesla V100-SMX2-16GB, Nvidia Tesla
V100-PCIE-32GB, or Nvidia L4-24GB GPUs. Sampling was performed on the latter
three GPUs due to their larger memory capacity. Our code execution utilized
OVH Public Cloud (ovh.com/) and Google Cloud resources
(https://console.cloud.google.com/). We tracked losses using Weights & Biases
(https://wandb.ai/). All machines and virtual machines featured 8 CPU cores
and 16GB of RAM. The IDE employed throughout the thesis was Visual Studio Code
(VSCode).

###  6.2 Quantitative Evaluation Results

We trained our models, plotted the training curves (refer to Appendix A.1),
and performed sampling on various datasets to evaluate our models.

|  | QM9
---|---|---
|  |  Real, $1\leq|V|\leq 9$
| Method |  Val. w/o corr. (%)$\uparrow$ |  NSPDK$\downarrow$ |  FCD$\downarrow$ |  Validity (%)$\uparrow$ |  Uniqueness (%)$\uparrow$ |  Novelty (%)$\uparrow$ |  Time (s)$\downarrow$
Autoreg. |  GraphAF [67] | 67* | 0.020 | 5.268 |  100.00* | 94.51* | 88.83* | $2.52e^{3}$
GraphAF+FC | 74.43 | 0.021 | 5.625 | 100.00 | 88.64 | 86.59 | $2.55e^{3}$
GraphDF [68] | 82.67* | 0.063 | 10.816 |  100.00* | 97.62* | 98.10* | $5.35e^{4}$
GraphDF+FC | 93.88 | 0.064 | 10.928 | 100.00 | 98.58 | 98.54 | $4.91e^{4}$
One shot |  MoFlow [66] | 91.36 | 0.017 | 4.467 | 100.00 | 98.65 | 94.72 | 4.60
EDP-GNN [52] | 47.52 | 0.005 | 2.680 | 100.00 | 99.25 | 86.58 | $4.40e^{3}$
GraphEBM [89] | 8.22 | 0.030 | 6.143 |  100.00* | 97.90* | 97.01* | $3.71e^{1}$
GDSS [17] | 95.72 | 0.003 | 3.096 | 100.00 | 98.4 | 86.10 | $8.4e^{1}$
SGGM+SLD [51] | 97.35 | 0.004 | 2.593 | 100.00 | 99.41 | 97.49 | ?
GRAPHARM [50] | 90.25 | 0.002 | 1.22 | ? | 95.62 | 70.39 | $1.52e^{1}$
CCSD (Ours) | 92.74 | 0.002 | 2.682 | 100.00 | 98.01 | 77.78 | $5.2e^{3}$
CCSD Base (Ours) | 92.42 | 0.008 | 4.320 | 100.00 | 97.13 | 84.26 | $6.5e^{3}$
Table 6.1: Generation results on the QM9 dataset. Results are taken from the
same seed (42). The best results are highlighted in bold. Values denoted by *
are taken from the respective original papers. Values denoted by ? are
unknown. Other results are taken from Jo et al. [17] or, for our results and
GDSS, have been retrained and calculated by ourselves. Val. w/o corr. denotes
the Validity w/o correction metric, and values that do not exceed 50% are
underlined. Results are rounded to 3 or 4 digits.

|  | Ego-small | Community-small
---|---|---|---
|  |  Real, $4\leq|V|\leq 18$ |  Synthetic, $12\leq|V|\leq 19$
|  |  Deg.$\downarrow$ |  Clus.$\downarrow$ |  Orbit$\downarrow$ |  Avg.$\downarrow$ |  Deg.$\downarrow$ |  Clus.$\downarrow$ |  Orbit$\downarrow$ |  Avg.$\downarrow$
Autoreg. |  DeepGMG [88] | 0.040 | 0.100 | 0.020 | 0.053 | 0.220 | 0.950 | 0.400 | 0.523
GraphRNN [22] | 0.090 | 0.220 | 0.003 | 0.104 | 0.080 | 0.120 | 0.040 | 0.080
GraphAF [67] | 0.03 | 0.11 | 0.001 | 0.047 | 0.18 | 0.20 | 0.02 | 0.133
GraphDF [68] | 0.04 | 0.13 | 0.01 | 0.060 | 0.06 | 0.12 | 0.03 | 0.070
One shot |  GraphVAE [24] | 0.130 | 0.170 | 0.050 | 0.117 | 0.350 | 0.980 | 0.540 | 0.623
GNF$\dagger$ [90] | 0.030 | 0.100 | 0.001 | 0.044 | 0.200 | 0.200 | 0.110 | 0.170
EDP-GNN [52] | 0.052 | 0.093 | 0.007 | 0.051 | 0.053 | 0.144 | 0.026 | 0.074
GDSS [17] | 0.021 | 0.024 | 0.007 | 0.017 | 0.077 | 0.064 | 0.013 | 0.051
SGGM+SLD [51] | 0.014 | 0.019 | 0.007 | 0.013 | 0.035 | 0.071 | 0.006 | 0.037
GRAPHARM [50] | 0.019 | 0.017 | 0.010 | 0.015 | 0.034 | 0.082 | 0.004 | 0.04
CCSD (Ours) | 0.030 | 0.023 | 0.018 | 0.024 | 0.114 | 0.063 | 0.065 | 0.081
CCSD Base (Ours) | ? | ? | ? | ? | 0.053 | 0.052 | 0.040 | 0.048

|  | Enzymes-small | Grid-small
---|---|---|---
|  |  Real, $4\leq|V|\leq 12$ |  Synthetic, $4\leq|V|\leq 49$
|  |  Deg.$\downarrow$ |  Clus.$\downarrow$ |  Orbit$\downarrow$ |  Avg.$\downarrow$ |  Deg.$\downarrow$ |  Clus.$\downarrow$ |  Orbit$\downarrow$ |  Avg.$\downarrow$
One shot | GDSS | 0.133 | 0.147 | 0.008 | 0.096 | 0.013 | 0.041 | 0.008 | 0.021
CCSD (Ours) | 0.234 | 0.155 | 0.013 | 0.134 | ? | ? | ? | ?
CCSD Base (Ours) | 0.129 | 0.329 | 0.030 | 0.163 | ? | ? | ? | ?

Table 6.2: Generation results on the generic graph datasets. Results are taken
from the same seed (42). We report the MMD distances between the test datasets
and generated graphs. The best results are highlighted in bold (the smaller
the better). Values denoted by ? are unknown. In the case of our models, it
means that we ran out of RAM during the sampling procedure. The results are
taken from Jo et al. [17] or, for our results and GDSS, have been retrained
and calculated by ourselves. $\dagger$ indicates unreproducible results.
Results are rounded to 3 or 4 digits.

|  | Ego-small | Community-small
---|---|---|---
|  |  Real, $4\leq|V|\leq 18$ |  Synthetic, $12\leq|V|\leq 19$
|  |  Rank-2$\downarrow$ |  Hodge.$\downarrow$
One shot | GDSS | ? | ?
CCSD (Ours) | 0.314 | 0.668
CCSD Base (Ours) | ? | 0.714

|  | Enzymes-small
---|---|---
|  |  Real, $4\leq|V|\leq 12$
|  |  Rank-2$\downarrow$ |  Hodge.$\downarrow$
One shot | GDSS | 0.017 | 0.558
CCSD (Ours) | 0.361 | 0.905
CCSD Base (Ours) | 0.345 | 0.783

|  | QM9
---|---|---
|  |  Real, $1\leq|V|\leq 9$
|  |  Rank-0$\downarrow$ |  Rank-1$\downarrow$ |  Rank-2$\downarrow$ |  Avg.$\downarrow$ |  Hodge.$\downarrow$
One shot | GDSS | 0.004 | 0.002 | 1.049 | 0.352 | ?
CCSD (Ours) | 0.0004 | 0.0023 | 1.3912 | 0.4646 | ?
CCSD Base (Ours) | 0.0002 | 0.0076 | 0.0129 | 0.0069 | ?

Table 6.3: Generation higher-order metrics results on all datasets. Results
are taken from the same seed (42). We report the MMD distances between the
test datasets and generated objects. The best results are highlighted in bold
(the smaller the better). Values denoted by ? are unknown. In the case of our
models, it means that we ran out of RAM during the sampling procedure. All the
metrics have been calculated by ourselves. - indicates that it is not relevant
for this dataset as there are no features attached for the corresponding
dimension. Results are rounded to 3 or 4 digits.

In the evaluation of our models on the QM9 dataset (Table 6.1), CCSD
demonstrates robust performance across various metrics, aligning closely with
state-of-the-art methods. Notably, our method excels in the NSPDK MMD measure,
indicating that it effectively generates molecular structures that closely
resemble the original distribution. Moreover, our framework exhibits high
Uniqueness and Novelty scores, suggesting its capability to also produce novel
molecular graphs while maintaining diversity and avoiding excessive similarity
between generated molecules.

For graph generation tasks (Table 6.2), both CCSD and CCSD Base consistently
yield promising results in both real and synthetic datasets. CCSD stands out
across all datasets, achieving impressive metric scores and rivalling other
methods. Specifically, for the Community small dataset, our two models achieve
state-of-the-art performance on the Clustering metric, beating the two latest
graph generation models recently published, SGGM+SLD and GRAPHARM, which
dominate the leaderboard. These results suggest that our approaches
effectively capture the inherent structural characteristics of graphs.

In the higher-order metrics assessment (Table 6.3), our approach consistently
performs well compared to GDSS. For the GDSS method, we generated graphs and
molecules and then lifted them into combinatorial complexes to calculate these
higher-order metrics for graph generation methods. This performance extends to
the QM9 dataset, where CCSD Base attains exceptionally low values across all
metrics, including rank-0, rank-1, rank-2, average, and the Hodge Laplacian
spectrum metric. These results underscore CCSD’s ability to generate objects
that closely align with the ground truth distribution.

To measure the quality of our samples, we also employed the Tanimoto
similarity, based on Morgan fingerprints obtained using the RDKit library [92]
with 1024 bits and a radius of 2 as suggested in Jo et al. [17] (see
Subsection 5.1.3 for a more detailed definition). We calculated the average
Tanimoto similarity of the generated molecules compared to all the training
molecules. As depicted in Table 6.4, CCSD has a higher similarity, thus
demonstrating its capability to generate molecules that closely match the
structural characteristics of training molecules, while other baseline models
including GDSS tend to generate molecules that could deviate from the training
distribution.

Tanimoto Similarity $\uparrow$
---
CCSD (Ours) | 0.564
GDSS | 0.502
Table 6.4: Tanimoto similarity on the QM9 dataset. We compared the average
Tanimoto similarity across the generated molecules by comparing their
fingerprints to the entire training dataset. The best results are highlighted
in bold (the larger the better). Results are rounded to 3 digits.

Overall, our framework excels in capturing target distributions, with
noteworthy performance in the NSPDK MMD metric for molecule generation,
surpassing other all the existing approaches. More importantly, our framework
performs well even despite its generality, as it is capable of generating not
only graphs and molecules but also higher-order topological structures. This
broader scope sets CCSD apart from competing approaches, positioning it as a
versatile and potent tool for diverse generative AI applications.

###  6.3 Qualitative Results

In this section, we present visualizations of the generated combinatorial
complexes for both generic graph generation tasks and molecule generation
tasks. These visualizations include the underlying graphs of the generated
combinatorial complexes.

####  6.3.1 Molecule Generation

In Figure 6.3, we showcase both the molecules generated by CCSD and a
selection of original molecules from the QM9 dataset. Figure 6.4 offers a
unique perspective by representing these molecules as hypergraphs, thus
illustrating the rings depicted as rank-2 cells. A resemblance is evident
between the generated molecules and the original distribution.

Figure 6.1: Original QM9 molecular dataset.
Figure 6.2: Generated QM9 molecules using CCSD.

Figure 6.3: Visualization of the combinatorial complexes (represented as
graphs) from the QM9 dataset and the generated objects of CCSD. Figure 6.4:
Visualization of the same combinatorial complexes generated via CCSD in Figure
6.3, represented as hypergraphs to visualize the rank-2 cells. The plot has
been realised using the HyperNetX Python library [103].

####  6.3.2 Generic Graph Generation

Moving on to our generic graph generation tasks, we provide visualizations of
the underlying graphs, drawn from both the training datasets and the generated
combinatorial complexes produced by CCSD. The displayed graphs are randomly
chosen from their respective datasets, accompanied by essential information,
including the number of nodes ($n$, representing rank-0 cells), the number of
edges ($e$, signifying rank-1 cells), and the count of faces ($f$,
corresponding to rank-2 cells) for each combinatorial complex.

Figure 6.5: Original Ego small graph dataset.
Figure 6.6: Generated Ego small objects using CCSD.

Figure 6.7: Visualization of the combinatorial complexes (represented as
graphs) from the Ego small dataset and the generated objects of CCSD.

Figure 6.8: Original Community small graph dataset.
Figure 6.9: Generated Community small objects using CCSD.

Figure 6.10: Visualization of the combinatorial complexes (represented as
graphs) from the Community small dataset and the generated objects of CCSD.

Figure 6.11: Original Enzymes small graph dataset.
Figure 6.12: Generated Enzymes small objects using CCSD.

Figure 6.13: Visualization of the combinatorial complexes (represented as
graphs) from the Enzymes small dataset and the generated objects of CCSD.

Likewise, it is clear that the graphs extracted from the generated
combinatorial complexes are similar to the original graph distribution.

Figure 6.14: Original Grid small graph dataset.

##  Chapter 7 Conclusion

In this concluding chapter, we address several key aspects of our work. We
begin by providing a concise summary of our achievements in Section 7.1. Then,
we present the ethical considerations that have guided our research in Section
7.2. Following that, we recognize and discuss the limitations of our work in
Section 7.3. Lastly, we offer valuable insights and directions for future
research in Section 7.4.

###  7.1 Summary of Achievements

Our work, despite its few limitations, stands as a significant contribution to
the fields of generative AI and topological deep learning. The key
achievements of this endeavour are the following. Firstly, we have
successfully designed and implemented CCSD, a score-based diffusion model
tailored for generating combinatorial complexes. This innovative approach
demonstrates promising outcomes across diverse datasets, encompassing tasks
such as molecule and graph generation. Additionally, we have introduced novel
objects, layers, and score neural network models that extend the horizons of
generative AI by enabling the generation of more abstract structures than
previously explored, thus pioneering Generative Topological Deep Learning.
Furthermore, we have devised new metrics specifically designed to evaluate the
quality of generated combinatorial complexes. Lastly, we have consolidated
these accomplishments into a robust Python library, poised to catalyze
advancements in the field.

###  7.2 Ethical Considerations

In terms of ethical considerations, our work is conducted in alignment with
established ethical guidelines and principles. It centres on the development
of a diffusion model for generating combinatorial complexes and poses no
direct ethical concerns. Our research solely aims at contributing to the
advancement of scientific knowledge and does not involve human or animal
subjects. Furthermore, the use of datasets does not raise any data-sharing
issues. We have diligently credited the authors of external arguments and
sources of inspiration, and we have acknowledged code that is not our own or
has been inspired by others’ work. Lastly, a strong emphasis has been placed
on clarity, both in the thesis and the code, to facilitate reproducibility.

###  7.3 Limitations

Our work harnesses diffusion models, a potent tool that also comes with its
set of limitations [39]. Firstly, one cannot dismiss the fact that the
assumption of iteratively denoising random noise, central to diffusion models,
probably does not align with how humans model their environment or analyze it.
Therefore, the approach of generating samples from noise and then denoising
iteratively might not fully encapsulate the intricate structures and
complexities inherent in real-world data.

In the context of Variational Diffusion Models, the interpretability of latent
representations is also a notable limitation. Unlike Variational Autoencoders,
where the encoder is optimized to yield structured latent spaces with the KL
Divergence term, VDM employs predetermined linear Gaussian encoders at each
time step. Consequently, intermediate latent representations in VDM remain as
noisy variants of the original input, impeding the model’s capacity to capture
meaningful and interpretable latent structures. More specific to our score-
based approach, interpreting and visualizing score functions related to graphs
or other complex objects pose considerable challenges [30, 22], let alone
partial score functions. Another constraint inherent to MHVAE and,
consequently, VDM, pertains to the latent space dimensionality. By confining
latent representations to the original input’s dimensionality, the model’s
potential to grasp higher-level abstractions, learn meaningful and compressed
representation, and generate diverse and expressive samples may be
compromised.

From a computational perspective, sampling from a diffusion model is
expensive, thus diminishing the practicality and scalability of the proposed
approach. Specifically, for graphs data, as the number of nodes grows, the
cost of generating a new graph through diffusion increases dramatically as
highlighted in some papers such as Vignac et al. [29]. More specific to our
work, the large search space associated with the higher-order incidence
matrices of combinatorial complexes exacerbates this already-known limitation.

Speaking of applications, most of the existing graph datasets primarily focus
on 2D data and/or artificial data, potentially inadequately representing
meaningful tasks such as learning from molecular datasets. This dearth of
representative graph generation datasets hinders the evaluation and
application of diffusion models in this domain [14]. Moreover, to the best of
our knowledge, there is no natural combinatorial complexes dataset available,
making our approach more difficult to benchmark.

Lastly, the intricate, irregular structures of graphs, characterized by
varying numbers of nodes and edges and heterogeneous properties, present
hurdles in designing effective diffusion models capable of capturing their
dynamics [104]. Accommodating the idiosyncratic structure of graphs within the
diffusion process calls for innovative and complex techniques.

###  7.4 Future Work

Looking to the future, several avenues for further research emerge. Firstly,
the exploration of new application domains that can leverage our combinatorial
complex modelling framework, such as mesh generation, could lead to promising
results. This includes exploring applications based on what we defined along
our framework: the conditional sampling of CCs (Subsection 4.2.2), imputation
of CCs (Subsection 4.2.3), and the generation with alternative training
objectives through penalization/regularization (Subsection 4.2.4). Extensive
research efforts could also be dedicated to conceiving new score network
models, accompanied by rigorous mathematical investigations to ensure their
permutation and equivariant properties. Furthermore, the development of
additional evaluation metrics tailored for generative topological deep
learning could substantially enhance the field’s progress. From an
implementation perspective, optimizing the pipeline to address the sparsity of
incidence matrices could also greatly increase the applicability of our
approach as it would increase scalability and potentially result in
accelerated computations and reduced RAM utilization.

## References

  * [1] Petar Veličković  “Everything is Connected: Graph Neural Networks”  arXiv, 2023  DOI: 10.48550/arXiv.2301.08210
  * [2] Michael M. Bronstein, Joan Bruna, Taco Cohen and Petar Veličković  “Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges”, 2021  arXiv:2104.13478 [cs.LG]
  * [3] Yilun Du and Igor Mordatch  “Implicit Generation and Modeling with Energy Based Models”  In _Advances in Neural Information Processing Systems_ 32 Curran Associates, Inc., 2019
  * [4] Florian Bordes, Sina Honari and Pascal Vincent  “Learning to Generate Samples from Noise through Infusion Training”, 2017  arXiv:1703.06975 [stat.ML]
  * [5] Anirudh Goyal, Nan Rosemary Ke, Surya Ganguli and Yoshua Bengio  “Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net”, 2017  arXiv:1711.02282 [stat.ML]
  * [6] Chitwan Saharia et al.  “Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding”, 2022  arXiv:2205.11487 [cs.CV]
  * [7] Aditya Ramesh et al.  “Zero-Shot Text-to-Image Generation”, 2021  arXiv:2102.12092 [cs.CV]
  * [8] Robin Rombach et al.  “High-Resolution Image Synthesis with Latent Diffusion Models”, 2022  arXiv:2112.10752 [cs.CV]
  * [9] Nanxin Chen et al.  “WaveGrad: Estimating Gradients for Waveform Generation”, 2020  arXiv:2009.00713 [eess.AS]
  * [10] Zhifeng Kong et al.  “DiffWave: A Versatile Diffusion Model for Audio Synthesis”, 2021  arXiv:2009.09761 [eess.AS]
  * [11] Thomas Segré  “A summary of the major contributions in score-based generative modeling”, 2022  DOI: 10.13140/RG.2.2.27162.72649
  * [12] Ruojin Cai et al.  “Learning Gradient Fields for Shape Generation”, 2020  arXiv:2008.06520 [cs.CV]
  * [13] Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev and Artem Babenko  “TabDDPM: Modelling Tabular Data with Diffusion Models”, 2022  arXiv:2209.15421 [cs.LG]
  * [14] Mengchun Zhang et al.  “A Survey on Graph Diffusion Models: Generative AI in Science for Molecule, Protein and Material” arXiv:2304.01565 [cs], 2023  DOI: 10.13140/RG.2.2.26493.64480
  * [15] Mathilde Papillon, Sophia Sanborn, Mustafa Hajij and Nina Miolane  “Architectures of Topological Deep Learning: A Survey on Topological Neural Networks”, 2023  arXiv:2304.10031 [cs.LG]
  * [16] Mustafa Hajij et al.  “Topological Deep Learning: Going Beyond Graph Data”, 2023
  * [17] Jaehyeong Jo, Seul Lee and Sung Ju Hwang  “Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations”, 2022  arXiv:2202.02514 [cs.LG]
  * [18] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan and Surya Ganguli  “Deep Unsupervised Learning using Nonequilibrium Thermodynamics”, 2015  arXiv:1503.03585 [cs.LG]
  * [19] Jonathan Ho, Ajay Jain and Pieter Abbeel  “Denoising Diffusion Probabilistic Models”, 2020  arXiv:2006.11239 [cs.LG]
  * [20] Jonathan Ho, Ajay Jain and Pieter Abbeel  “Denoising Diffusion Probabilistic Models”  In _Advances in Neural Information Processing Systems_ 33 Curran Associates, Inc., 2020, pp. 6840–6851
  * [21] Adrien Carrel  “CCSD - Combinatorial Complex Score-based Diffusion model using stochastic differential equations.”, 2023  URL: https://github.com/AdrienC21/CCSD
  * [22] Jiaxuan You et al.  “GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models”, 2018  arXiv:1802.08773 [cs.LG]
  * [23] Mariya Popova, Mykhailo Shvets, Junier Oliva and Olexandr Isayev  “MolecularRNN: Generating realistic molecular graphs with optimized properties”, 2019  arXiv:1905.13372 [cs.LG]
  * [24] Martin Simonovsky and Nikos Komodakis  “GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders”, 2018  arXiv:1802.03480 [cs.LG]
  * [25] Nicola De Cao and Thomas Kipf  “MolGAN: An implicit generative model for small molecular graphs”, 2022  arXiv:1805.11973 [stat.ML]
  * [26] Han Huang et al.  “GraphGDP: Generative Diffusion Processes for Permutation Invariant Graph Generation”, 2022  arXiv:2212.01842 [cs.LG]
  * [27] Minkai Xu et al.  “GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation”, 2022  arXiv:2203.02923 [cs.LG]
  * [28] Bowen Jing et al.  “Torsional Diffusion for Molecular Conformer Generation”  arXiv, 2023  DOI: 10.48550/arXiv.2206.01729
  * [29] Clement Vignac et al.  “DiGress: Discrete Denoising diffusion for graph generation”  arXiv, 2023  DOI: 10.48550/arXiv.2209.14734
  * [30] John Ingraham, Vikas Garg, Regina Barzilay and Tommi Jaakkola  “Generative Models for Graph-Based Protein Design”  In _Advances in Neural Information Processing Systems_ 32 Curran Associates, Inc., 2019
  * [31] Kilian Konstantin Haefeli, Karolis Martinkus, Nathanaël Perraudin and Roger Wattenhofer  “Diffusion Models for Graphs Benefit From Discrete State Spaces”, 2022  arXiv:2210.01549 [cs.LG]
  * [32] Yang Song et al.  “Score-Based Generative Modeling through Stochastic Differential Equations”, 2021  arXiv:2011.13456 [cs.LG]
  * [33] Tianze Luo, Zhanfeng Mo and Sinno Jialin Pan  “Fast Graph Generation via Spectral Diffusion”, 2022  arXiv:2211.08892 [cs.LG]
  * [34] W. Walters and Regina Barzilay  “Applications of Deep Learning in Molecule Generation and Molecular Property Prediction” Publisher: American Chemical Society  In _Accounts of Chemical Research_ 54.2, 2021, pp. 263–270  DOI: 10.1021/acs.accounts.0c00699
  * [35] Kenneth Atz, Francesca Grisoni and Gisbert Schneider  “Geometric Deep Learning on Molecular Representations”, 2021  arXiv:2107.12375 [physics.chem-ph]
  * [36] Alex Zhavoronkov  “Artificial Intelligence for Drug Discovery, Biomarker Development, and Generation of Novel Chemistry”  In _Molecular Pharmaceutics_ 15.10, 2018, pp. 4311–4313  DOI: 10.1021/acs.molpharmaceut.8b00930
  * [37] Cristian Bodnar et al.  “Weisfeiler and Lehman Go Cellular: CW Networks”, 2022  arXiv:2106.12575 [cs.LG]
  * [38] Manh Tuan Do, Se-eun Yoon, Bryan Hooi and Kijung Shin  “Structural Patterns and Generative Models of Real-world Hypergraphs”  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_ ACM, 2020  DOI: 10.1145/3394486.3403060
  * [39] Calvin Luo  “Understanding Diffusion Models: A Unified Perspective”  arXiv, 2022  DOI: 10.48550/arXiv.2208.11970
  * [40] David McAllester  “On the Mathematics of Diffusion Models” arXiv:2301.11108 [cs, math]  arXiv, 2023  DOI: 10.48550/arXiv.2301.11108
  * [41] Sitan Chen et al.  “Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions”, 2023  arXiv:2209.11215 [cs.LG]
  * [42] Bernt Øksendal  “Stochastic Differential Equations”, Universitext  Berlin, Heidelberg: Springer, 2003  DOI: 10.1007/978-3-642-14394-6
  * [43] Diederik P Kingma and Max Welling  “Auto-Encoding Variational Bayes”, 2022  arXiv:1312.6114 [stat.ML]
  * [44] Durk P Kingma et al.  “Improved Variational Inference with Inverse Autoregressive Flow”  In _Advances in Neural Information Processing Systems_ 29 Curran Associates, Inc., 2016
  * [45] Casper Kaae Sønderby et al.  “Ladder Variational Autoencoders”  In _Advances in Neural Information Processing Systems_ 29 Curran Associates, Inc., 2016
  * [46] Diederik Kingma, Tim Salimans, Ben Poole and Jonathan Ho  “Variational Diffusion Models”  In _Advances in Neural Information Processing Systems_ 34 Curran Associates, Inc., 2021, pp. 21696–21707
  * [47] Yann Lecun et al.  “A tutorial on energy-based learning”  In _Predicting structured data_ MIT Press, 2006
  * [48] Yang Song and Diederik P. Kingma  “How to Train Your Energy-Based Models”, 2021  arXiv:2101.03288 [cs.LG]
  * [49] Yang Song and Stefano Ermon  “Generative Modeling by Estimating Gradients of the Data Distribution”  In _Advances in Neural Information Processing Systems_ 32 Curran Associates, Inc., 2019  URL: https://proceedings.neurips.cc/paper_files/paper/2019/file/3001ef257407d5a371a96dcd947c7d93-Paper.pdf
  * [50] Lingkai Kong et al.  “Autoregressive Diffusion Model for Graph Generation”, 2023  arXiv:2307.08849 [cs.AI]
  * [51] Ling Yang, Zhilong Zhang, Wentao Zhang and Shenda Hong  “Score-Based Graph Generative Modeling with Self-Guided Latent Diffusion”, 2023  URL: https://openreview.net/forum?id=AykEgQNPJEK
  * [52] Chenhao Niu et al.  “Permutation Invariant Graph Generation via Score-Based Generative Modeling”, 2020  arXiv:2003.00638 [cs.LG]
  * [53] Chence Shi, Shitong Luo, Minkai Xu and Jian Tang  “Learning Gradient Fields for Molecular Conformation Generation”, 2021  arXiv:2105.03902 [cs.LG]
  * [54] B. Efron  “Tweedie’s Formula and Selection Bias”  In _Journal of the American Statistical Association_ 106.496, 2011, pp. 1602–1614  URL: http://www.jstor.org/stable/23239562
  * [55] Daniel Revuz and Marc Yor  “Martingales”  In _Continuous Martingales and Brownian Motion_ Berlin, Heidelberg: Springer Berlin Heidelberg, 1999, pp. 51–77  DOI: 10.1007/978-3-662-06400-9˙3
  * [56] Brian D.O. Anderson  “Reverse-time diffusion equation models”  In _Stochastic Processes and their Applications_ 12.3, 1982, pp. 313–326  URL: https://ideas.repec.org/a/eee/spapps/v12y1982i3p313-326.html
  * [57] Aapo Hyvarinen  “Estimation of Non-Normalized Statistical Models by Score Matching”  In _Journal of Machine Learning Research_ 6.24, 2005, pp. 695–709  URL: http://jmlr.org/papers/v6/hyvarinen05a.html
  * [58] Yang Song, Sahaj Garg, Jiaxin Shi and Stefano Ermon  “Sliced Score Matching: A Scalable Approach to Density and Score Estimation”  In _Proceedings of The 35th Uncertainty in Artificial Intelligence Conference_ 115, Proceedings of Machine Learning Research  PMLR, 2020, pp. 574–584  URL: https://proceedings.mlr.press/v115/song20a.html
  * [59] Gaspard Monge  “Mémoire sur la théorie des déblais et des remblais” OCLC: 51928110  Paris: De l’Imprimerie Royale, 1781
  * [60] Adrien Carrel  “Optimal transport applied to color transportation in image processing.”, 2019  URL: https://github.com/AdrienC21/optimal-transport-color-transportation
  * [61] Ine Melckenbeeck, Pieter Audenaert, Didier Colle and Mario Pickavet  “Efficiently counting all orbits of graphlets of any order in a graph using autogenerated equations”  In _Bioinformatics_ 34.8, 2017, pp. 1372–1380  DOI: 10.1093/bioinformatics/btx758
  * [62] Natasa Przulj, Derek G Corneil and Igor Jurisica  “Modeling interactome: scale-free or geometric?”  In _Bioinformatics_ 20.18 Oxford University Press, 2004, pp. 3508–3515
  * [63] Tomaz Hosevar and Janez Demsar  “A combinatorial approach to graphlet counting”  In _Bioinformatics_ 30.4 Oxford University Press, 2014, pp. 559–565
  * [64] Kristina Preuer et al.  “Frechet ChemNet Distance: A metric for generative models for molecules in drug discovery”, 2018  arXiv:1803.09518 [cs.LG]
  * [65] Fabrizio Costa and Kurt De Grave  “Fast Neighborhood Subgraph Pairwise Distance Kernel”  In _International Conference on Machine Learning_ , 2010  URL: https://api.semanticscholar.org/CorpusID:16262476
  * [66] Chengxi Zang and Fei Wang  “MoFlow: An Invertible Flow Model for Generating Molecular Graphs”  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining_ ACM, 2020  DOI: 10.1145/3394486.3403104
  * [67] Chence Shi et al.  “GraphAF: a Flow-based Autoregressive Model for Molecular Graph Generation”, 2020  arXiv:2001.09382 [cs.LG]
  * [68] Youzhi Luo, Keqiang Yan and Shuiwang Ji  “GraphDF: A Discrete Flow Model for Molecular Graph Generation”, 2021  arXiv:2102.01189 [cs.LG]
  * [69] John J. Irwin et al.  “ZINC: A Free Tool to Discover Chemistry for Biology” Publisher: American Chemical Society  In _Journal of Chemical Information and Modeling_ 52.7, 2012, pp. 1757–1768  DOI: 10.1021/ci3001277
  * [70] Michael T. Schaub et al.  “Signal processing on higher-order networks: Livin’ on the edge… and beyond”  In _Signal Processing_ 187 Elsevier BV, 2021, pp. 108149  DOI: 10.1016/j.sigpro.2021.108149
  * [71] Sergio Barbarossa and Stefania Sardellitti  “Topological Signal Processing Over Simplicial Complexes”  In _IEEE Transactions on Signal Processing_ 68 Institute of ElectricalElectronics Engineers (IEEE), 2020, pp. 2992–3007  DOI: 10.1109/tsp.2020.2981920
  * [72] Lek-Heng Lim  “Hodge Laplacians on Graphs”  In _SIAM Review_ 62.3, 2020, pp. 685–715  DOI: 10.1137/18M1223101
  * [73] Pascal Vincent  “A Connection Between Score Matching and Denoising Autoencoders”  In _Neural Computation_ 23, 2011, pp. 1661–1674  URL: https://api.semanticscholar.org/CorpusID:5560643
  * [74] S. Sarkka and A. Solin  “Applied Stochastic Differential Equations”, Institute of Mathematical Statistics Textbooks  Cambridge University Press, 2019
  * [75] Tianyu Pang et al.  “Efficient Learning of Generative Models via Finite-Difference Score Matching”, 2020  arXiv:2007.03317 [cs.LG]
  * [76] Yibo Li, Liangren Zhang and Zhenming Liu  “Multi-Objective De Novo Drug Design with Conditional Graph Generative Model”, 2018  arXiv:1801.07299 [q-bio.QM]
  * [77] Ricky T.. Chen, Yulia Rubanova, Jesse Bettencourt and David Duvenaud  “Neural Ordinary Differential Equations”, 2019  arXiv:1806.07366 [cs.LG]
  * [78] Geoffrey Roeder, Luke Metz and Diederik P. Kingma  “On Linear Identifiability of Learned Representations”, 2020  arXiv:2007.00810 [stat.ML]
  * [79] Geoffrey E. Hinton  “Connectionist Learning Procedures”  In _Artif. Intell._ 40, 1989, pp. 185–234  URL: https://api.semanticscholar.org/CorpusID:7840452
  * [80] Terence Sanger and Pallavi N. Baljekar  “The perceptron: a probabilistic model for information storage and organization in the brain.”  In _Psychological review_ 65 6, 1958, pp. 386–408  URL: https://api.semanticscholar.org/CorpusID:12781225
  * [81] Sergey Ioffe and Christian Szegedy  “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”, 2015  arXiv:1502.03167 [cs.LG]
  * [82] Thomas N. Kipf and Max Welling  “Semi-Supervised Classification with Graph Convolutional Networks”, 2017  arXiv:1609.02907 [cs.LG]
  * [83] Jinheon Baek, Minki Kang and Sung Ju Hwang  “Accurate Learning of Graph Representations with Graph Multiset Pooling”, 2021  arXiv:2102.11533 [cs.LG]
  * [84] Nicolas Keriven and Gabriel Peyré  “Universal Invariant and Equivariant Graph Neural Networks”, 2019  arXiv:1905.04943 [cs.LG]
  * [85] Chanon Thongprayoon, Lorenzo Livi and Naoki Masuda  “Embedding and Trajectories of Temporal Networks”  In _IEEE Access_ 11 Institute of ElectricalElectronics Engineers (IEEE), 2023, pp. 41426–41443  DOI: 10.1109/access.2023.3268030
  * [86] Guido Van Rossum and Fred L Drake Jr  “Python reference manual”  Centrum voor Wiskunde en Informatica Amsterdam, 1995
  * [87] ISO  “ISO IEC 14882:1998: Programming languages — C++”, 1998, pp. 732
  * [88] Yujia Li et al.  “Learning Deep Generative Models of Graphs”, 2018  arXiv:1803.03324 [cs.LG]
  * [89] Meng Liu, Keqiang Yan, Bora Oztekin and Shuiwang Ji  “GraphEBM: Molecular Graph Generation with Energy-Based Models”, 2021  arXiv:2102.00546 [cs.LG]
  * [90] Jenny Liu et al.  “Graph Normalizing Flows”  In _Advances in Neural Information Processing Systems_ 32 Curran Associates, Inc., 2019
  * [91] Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp and O. Lilienfeld  “Quantum chemistry structures and properties of 134 kilo molecules”  In _Scientific Data_ 1.1, 2014, pp. 140022  DOI: 10.1038/sdata.2014.22
  * [92] Greg Landrum  “RDKit: Open-Source Cheminformatics Software”, 2016  URL: https://github.com/rdkit/rdkit/releases/tag/Release_2016_09_4
  * [93] Prithviraj Sen et al.  “Collective Classification in Network Data”  In _AI Magazine_ 29.3, 2008, pp. 93  DOI: 10.1609/aimag.v29i3.2157
  * [94] Ida Schomburg et al.  “BRENDA, the enzyme database: Updates and major new developments”  In _Nucleic acids research_ 32, 2004, pp. D431–3  DOI: 10.1093/nar/gkh081
  * [95] Aric Hagberg, Pieter Swart and Daniel S Chult  “Exploring network structure, dynamics, and function using NetworkX”, 2008
  * [96] Tomaz Hocevar and Janez Demsar  “A combinatorial approach to graphlet counting”  In _Bioinformatics_ 30 4, 2014, pp. 559–65  URL: https://api.semanticscholar.org/CorpusID:33092354
  * [97] Renjie Liao et al.  “Efficient Graph Generation with Graph Recurrent Attention Networks”, 2020  arXiv:1910.00760 [cs.LG]
  * [98] Leslie O’Bray, Max Horn, Bastian Rieck and Karsten Borgwardt  “Evaluation Metrics for Graph Generative Models: Problems, Pitfalls, and Practical Solutions”, 2022  arXiv:2106.01098 [cs.LG]
  * [99] Yang Song and Stefano Ermon  “Improved Techniques for Training Score-Based Generative Models”  In _Advances in Neural Information Processing Systems_ 33 Curran Associates, Inc., 2020, pp. 12438–12448
  * [100] Dávid Bajusz, Anita Rácz and Károly Héberger  “Why is Tanimoto index an appropriate choice for fingerprint-based similarity calculations?”  In _Journal of Cheminformatics_ 7, 2015  DOI: 10.1186/s13321-015-0069-3
  * [101] H.. Morgan  “The Generation of a Unique Machine Description for Chemical Structures-A Technique Developed at Chemical Abstracts Service.” Publisher: American Chemical Society  In _Journal of Chemical Documentation_ 5.2, 1965, pp. 107–113  DOI: 10.1021/c160017a018
  * [102] Adam Paszke et al.  “PyTorch: An Imperative Style, High-Performance Deep Learning Library”, 2019  arXiv:1912.01703 [cs.LG]
  * [103] Brenda Praggastis et al.  “pnnl/HyperNetX”
  * [104] Petar Veličković et al.  “Graph Attention Networks”, 2018  arXiv:1710.10903 [stat.ML]

##  Appendix A Appendix

###  A.1 Learning curves

This section includes the learning curves we obtained for each dataset,
presenting the training and testing losses for each partial score function. As
a reminder, the loss is measured as the $L2$ distance between the predicted
score and the actual score.

####  A.1.1 QM9

Figure A.1: Train losses QM9.

Figure A.2: Test losses QM9.

####  A.1.2 Ego small

Figure A.3: Train losses Ego small.

Figure A.4: Test losses Ego small.

####  A.1.3 Community small

Figure A.5: Train losses Community small.

Figure A.6: Test losses Community small.

####  A.1.4 Enzymes small

Figure A.7: Train losses Enzymes small.

Figure A.8: Test losses Enzymes small.

####  A.1.5 Grid small

Figure A.9: Train losses Grid small.

Figure A.10: Test losses Grid small.


# What drives the $M_{\ast}$$-$SFR relation turning over at high masses? The
role of bulges

Zhizheng Pan11affiliation: Key laboratory for research in galaxies and
cosmology, Shanghai Astronomical Observatory, Chinese Academy of Science, 80
Nandan Road, Shanghai, 200030, China , Xianzhong Zheng 44affiliation: Purple
Mountain Observatory, Chinese Academy of Sciences, 2 West-Beijing Road,
Nanjing 210008, China , Weipeng Lin 11affiliation: Key laboratory for research
in galaxies and cosmology, Shanghai Astronomical Observatory, Chinese Academy
of Science, 80 Nandan Road, Shanghai, 200030, China 55affiliation: School of
Astronomy and Space Science, Sun Yat-Sen University, Guangzhou, 510275, China
, Jinrong Li22affiliation: Center of Astrophysics, University of Science and
Technology of China, Jinzhai Road 96, Hefei 230026, China 33affiliation: Key
Laboratory for Research in Galaxies and Cosmology, USTC, CAS, China , Jing
Wang66affiliation: CSIRO Astronomy & Space Science, Australia Telescope
National Facility, PO Box 76, Epping, NSW 1710, Australia ,Lulu
Fan77affiliation: Institute of Space Sciences, Shandong University, Weihai,
264209, China , Xu Kong22affiliation: Center of Astrophysics, University of
Science and Technology of China, Jinzhai Road 96, Hefei 230026, China
33affiliation: Key Laboratory for Research in Galaxies and Cosmology, USTC,
CAS, China<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

It is unclear whether bulge growth is responsible for the flattening of the
star formation main sequence (MS) at the high mass end. To investigate the
role of bulges in shaping the MS, we compare the NUV$-r$ color between the
central ($r<R_{50}$) and outer regions for a sample of 6401 local star-forming
galaxies. The NUV$-r$ color is a good specific star formation rate indicator.
We find that at $M_{\ast}<10^{10.2}M_{\sun}$, the central NUV$-r$ is on
average only $\sim$ 0.25 mag redder than the outer NUV$-r$. Above
$M_{\ast}=10^{10.2}M_{\sun}$, the central NUV$-r$ becomes systematically much
redder than the outer NUV$-r$ for more massive galaxies, indicating that the
central bulge is more evolved at the massive end. When dividing the galaxies
according to their Sérsic index $n$, we find that galaxies with $n$¿2.0 tend
to be redder in the central NUV$-r$ color than those with $n$¡2.0, even at
fixed B/T and $M_{\ast}$. This suggests that star formation in bulges is more
strongly dependent on $n$ (or central mass density) than on B/T. Finally, we
find that the fraction of galaxies with $n$¿2.0 rapidly increases with
$M_{\ast}$ at $M_{\ast}>10^{10.2}M_{\sun}$, which is consistent with the
turning over of the MS at the same transition mass. We conclude that the
increasing fraction of low-sSFR dense bulges in $M_{\ast}>10^{10.2}M_{\sun}$
galaxies, rather than increasing B/T, is responsible for the flattened slope
of the $M_{\ast}$$-$SFR relation at high masses.

###### Subject headings:

galaxies: evolution – galaxies: star formation

##  1\. Introduction

For star-forming galaxies (SFGs), a tight correlation is found between stellar
mass ($M_{\ast}$) and star formation rate (SFR), which is commonly referred to
the star formation main sequence (MS) (Brinchmann et al., 2004; Daddi et al.,
2007; Elbaz et al., 2007; Guo et al., 2013). The MS can be described by a
single pow-law formulated by log$~{}SFR$$\propto\alpha$$\rm
log$$~{}M_{\ast}+\beta$, where $\alpha$ is between 0.6 and 1.0 (Peng et al.,
2010; Karim et al., 2011; Whitaker et al., 2012). Recently, the form of the MS
is found to be better fitted by two power-laws. Below a transition mass of
$M_{\ast}\sim 10^{10.2}M_{\sun}$, the MS has a steep slope of $\alpha\sim
1.0$. Above the transition mass, the slope becomes flattened, with
$\alpha=[0.2,0.8]$, depending on redshift $z$ (Whitaker et al., 2014). The
physical mechanisms that drive the curvature of the MS at $M_{\ast}\sim
10^{10.2}M_{\sun}$ are still unclear.

Among the proposed mechanisms responsible for the curvature of the MS, galaxy
bulge growth is most favored (Whitaker et al., 2014; Lee et al., 2015;
Schreiber et al., 2015). Observationally, more massive galaxies are indeed
more bulgy and tend to have higher bulge-to-total mass ratio (B/T). As shown
in previous works, the central bulges of massive galaxies are generally old
and dead (Pérez et al., 2013; Pan et al., 2014). Given that the bulge
component contributes little to the total SFR of a galaxy, the specific star
formation rate (sSFR, termed as sSFR=SFR/$M_{\ast}$) of a high B/T galaxy
should be lower than that of a disk dominated galaxy, even they have the same
$M_{\ast}$. This idea is supported by the work of Abramson et al. (2014), who
found that when accounting for disk/bulge decomposition, the disk mass
normalized SFR ($\rm sSFR_{\rm disk}$, formulated by $\rm sSFR_{\rm
disk}$=SFR/$M_{\rm disk}$, where $M_{\rm disk}$ is the stellar mass of disk
component) shows a weak dependence on $M_{\ast}$.

Abramson et al. (2014) argues that the star formation in galaxies is mainly
contributed from galactic disk components. However, the details of how bulges
affect the star formation activities in galaxies remain to be explored. To
better understand the role of bulges in shaping the MS relation, in this
Letter, we use the $GALEX$ (Martin et al., 2005) and SDSS (York et al., 2000)
data to investigate the recent star formation in the central and outer regions
of a large local SFGs sample. Throughout this Letter, we assume a concordance
$\Lambda$CDM cosmology with $\Omega_{\rm m}=0.3$, $\Omega_{\rm\Lambda}=0.7$,
$H_{\rm 0}=70$ $\rm km~{}s^{-1}$ Mpc-1, and a Kroupa (2001) IMF.

Figure 1.— a): the color–magnitude diagram. The star forming galaxies (SFGs)
are selected by the blue-dashed line. b): The $f_{\rm central}$ distribution
of SFGs, where $f_{\rm central}$ was defined as $f_{\rm central}=\rm Flux_{\rm
r=6\farcs 0}/\rm Flux_{\rm total}$. The GALEX NUV and SDSS $r$-band is shown
in blue and red histograms, respectively. Panel c): the B/T–$M_{\ast}$ diagram
of SFGs. Panel d): the B/T–Sérsic index $n$ diagram of SFGs.

##  2\. Method and data used

To assess the role of bulges in shaping the MS, a straightforward way is to
derive the sSFR of the bulge component ( $\rm sSFR_{\rm bulge}$). Since the
direct measure of $\rm sSFR_{\rm bulge}$ is quite difficult, in this work we
use the NUV$-r_{\rm bulge}$ color as a $\rm sSFR_{\rm bulge}$ indicator. The
NUV$-r$ is more tightly correlated with sSFR than the $u-r$ color, especially
at the low sSFR end (Salim, 2014). This is not surprised because the UV
luminosity $L_{\rm UV}$ is an SFR indicator (e.g., Kennicutt, 1998) and
NUV$-r$ is thus a proxy of SFR/$L_{\rm r}$, where $L_{\rm r}$ is the $r$-band
luminosity. Thus the NUV$-r$ actually tells one the luminosity weighted sSFR.

Traditionally, a bulge+disk decomposition is required if one wants to derive
the NUV$-r_{\rm bulge}$. Unfortunately, the $\it GALEX$ imaging has a
relatively low resolution (The _GALEX_ NUV image has a resolution of 1
pixel=1$\farcs$5 and a point spread function (PSF) with full width at half-
maximum (FWHM)=5$\farcs$3). Therefore a reliable bulge+disk decomposition
procedure can be done only for galaxies with large angular sizes, which will
yield a very small sample size. To investigate a large sample, in this work we
instead measure the NUV$-r$ color of the central region of a galaxy
(NUV$-r_{\rm central}$) as the representative of NUV$-r_{\rm bulge}$ of the
galaxy. We refer the central region to be $R_{\rm central}\approx R_{50}$,
where $R_{50}$ is the radius enclosing 50% of the SDSS $r$-band petrosian
flux. By doing this, one can simply perform aperture photometry to obtain
NUV$-r_{\rm central}$ for a galaxy sample with similar angular sizes.

Following Wang et al. (2010), we have constructed a UV-optical matched
photometric catalog. This catalog contains about 220,000 galaxies with uniform
photometric measurements on the resolution and PSF-matched _GALEX_ +SDSS
images, which are cross-matched between the SDSS DR8 (Aihara et al., 2011)
spectroscopic galaxies and the _GALEX_ GR6 database. For each galaxy, the
fluxes were measured using 5 different apertures, with $r=$[1.5, 3.0, 6.0,
9.0, 12.0] ″, in the FUV, NUV, $u,g,r,i,z$ bands. We also measure the total
magnitude of the galaxies with SExtractor (Bertin& Arnouts, 1996). All the
magnitudes have been corrected for galactic extinction using the galactic dust
map (Schlegel, Finkbeiner & Davis, 1998). The details of data reduction
procedure can be found in Wang et al. (2010) and Pan et al. (2014).

Figure 2.— a):The relation between NUV$-r_{\rm outer}$ and $M_{\ast}$ for
SFGs, where NUV$-r_{\rm outer}$ is the color index measured outside the
$r=$6$\farcs$0 aperture. The color represent galaxies with different
$M_{\ast}$. The median values of NUV$-r_{\rm outer}$ values are denoted in
black open circles. The blue dashed line denotes our SFGs selection criterion.
b): The relation between NUV$-r_{\rm central}$ and $M_{\ast}$. Median values
are shown in black filled circles. c): [NUV$-r_{\rm central}$]–[NUV$-r_{\rm
outer}$] as a function function of $M_{\ast}$. Median values are shown in
black crosses. d): the normalized [NUV$-r_{\rm central}$]–[NUV$-r_{\rm
outer}$] distributions. The colors represent different mass bins.

To ensure that the measured central NUV$-r$ color is not significantly
affected by the relatively poor resolution of the _GALEX_ images, we limit the
sample galaxies with $R_{50}\sim 6\farcs 0$. Our sample selection criteria are
as follows:

(1) minor$-$major axis ratio $b/a>0.5$, to minimize dust reddening effect;

(2) 4$\farcs$0¡$R_{50}$¡8$\farcs$0, to ensure that the chosen central aperture
encloses $\sim 50$% of the total flux of a galaxy;

(3) $z=[0.005,0.1]$, where $z$ is the SDSS spectroscopic redshift;

(4) $m_{\rm NUV}<m_{\rm limit}$, where $m_{\rm limit}=23.0~{}$mag is the
limited magnitude in the NUV band;

(5) stellar mass $M_{\ast}>10^{9.0}~{}M_{\sun}$.

The selected sample contains 8200 galaxies. We note that this sample is not
volume-completed and our conclusions do not rely on the selection completeness
in the volume. The stellar mass $M_{\ast}$ used in this Letter was drawn from
the JHU/MPA database 111http://www.mpa-garching.mpg.de/SDSS/DR7. Since all
AGNs in our sample are narrow-line (obscured) AGNs, there should be no
contribution from AGN continuum to the measured NUV$-r$ index.

The properties of our sample galaxies are shown in Figure 1. panel a) is the
color magnitude diagram. We select SFGs with NUV$-r$¡$0.6\times M_{\ast}-2.4$
(the blue dashed line), where $M_{\ast}$ is logarithm stellar mass. In total,
a sample of 6401 SFGs is assembled. Panel b) shows the flux fraction
distribution enclosed in the central $R=6.\arcsec 0$ aperture for the selected
SFGs. One can see that the central aperture encloses $\sim 50$% of the
$r-$band total flux. In panel c), we show the bulge-to-total ratio B/T as a
function of $M_{\ast}$. The B/T is from the disk+bulge decomposition on the
SDSS $r$-band imaging (Simard et al., 2011). As can be seen, the B/T
distribution is significantly different at $M_{\ast}<10^{10.2}M_{\sun}$ and
$M_{\ast}>10^{10.2}M_{\sun}$, in the sense that more massive galaxies have
higher B/T values. However, the majority of SFGs have a B/T¡0.5. Therefore,
the chosen aperture of $r_{\rm central}\simeq R_{50}$ is enough to separate
the bulges from the disk components for the SFGs. Panel d) presents B/T as a
function of the Sérsic index $n$ of the galaxy. Here $n$ is fitted with a pure
Sérsic model (see Simard et al. (2011) for details). As also shown in Simard
et al. (2011), B/T increases with $n$. Below we present our analysis results
for the sample of selected SFGs.

##  3\. The dependence of NUV$-r_{\rm central}$ and NUV$-r_{\rm outer}$ on
$M_{\ast}$

Figure 2 presents the outer and central NUV$-r$ for our sample of SFGs. Here
the outer/central region of a galaxy refers to the part out/within a circular
aperture of radius $R=6\farcs 0$. The SFGs are divided in 6 mass bins. As one
can see from panel a), the NUV$-r_{\rm outer}$ is redder for more massive
galaxies. However, no turning over is seen at log($M_{\ast}/M_{\sun}$)=10.2.
In panel b), we show the [NUV$-r_{\rm central}$]–$M_{\ast}$ diagram. For
comparison, we overplot the median values of the NUV$-r_{\rm outer}$ as well.
It is clear that the median value of NUV$-r_{\rm central}$ is only slightly
redder than that of NUV$-r_{\rm outer}$ at log($M_{\ast}/M_{\sun}$)¡10.2,
while the discrepancy between the two becomes larger at higher masses. Panel
c) shows the discrepancy ([NUV$-r_{\rm central}$]–[NUV$-r_{\rm outer}$])
between the two as a function of $M_{\ast}$. Note that we do not apply the
internal extinction correction for the NUV$-r$ measurements. If the extinction
correction is similar for these two color indices, then [NUV$-r_{\rm
central}$]–[NUV$-r_{\rm outer}$] reflects the intrinsic difference in color,
and then the difference in sSFR. As one can see, this relation bends at
log($M_{\ast}/M_{\sun}$)$\sim$10.2, which is similar to the MS. In panel d),
we show the [NUV$-r_{\rm central}$]–[NUV$-r_{\rm outer}$] distribution.
Strikingly, the three mass bins with log($M_{\ast}/M_{\sun}$)¡10.2 exhibit
similar distributions peaked at [NUV$-r_{\rm central}$]–[NUV$-r_{\rm
outer}$]$\sim$0.25 mag. Above log($M_{\ast}/M_{\sun}$)=10.2, the distributions
start to shift redward.

Figure 3.— The [NUV$-r_{\rm central}$]–[NUV$-r_{\rm outer}$] distributions for
the log($M_{\ast}/M_{\sun}$)¿10.2 SFGs with $n$¡2.0 (upper panels) and $n$¿2.0
(upper panels) . From left to right, we show the result for different mass
bins. In each mass bin, we further divide the SFGs into 3 small bins according
to their B/T values. SFGs of the lowest B/T values are shown in thinnest
lines, while those with highest B/T are shown in thickest lines. In each bin,
the number of the SFGs are marked.

In short, the results presented in Figure 2 support the argument that the bend
of the MS is indeed attributed to the presence of red bulges in massive
galaxies. This findings also help one in better interpreting the increasing
intrinsic scatters of the MS with $M_{\ast}$ (Guo et al., 2013, 2015). Based
on the large sample, this result should be reliable in a statistical sense.
Interestingly, we find the trend that NUV$-r_{\rm central}$ is only 0.25 mag
redder than NUV$-r_{\rm outer}$ holds for a mass range of
log($M_{\ast}/M_{\sun}$)¡10.2. This indicates that the central regions of less
massive galaxies still have quite high $\rm sSFR_{\rm central}$. This finding
suggests that the mechanisms driving bulge growth of low-mass galaxies is
similar to those regulating disk growth. In other words, bulges in low mass-
SFGs tend to be psuedo type. Given this, in the next section, we will
investigate the correlations of $n$ with NUV$-r$ as well as B/T.

##  4\. What leads to dead bulges?

Figure 2 supports that the red bulges of log($M_{\ast}/M_{\sun})>10.2$
galaxies are responsible (at least partly) for the turning over of the MS
relation. In this section we address how B/T and $n$ correlate with the read-
and-dead of central bulges.

We first divide the SFGs into two categories with $n$¡2.0 and $n$¿2.0. In each
$n$ bin, we investigate the behavior of [NUV$-r_{\rm central}$]–[NUV$-r_{\rm
outer}$] as a function of B/T. We have checked that galaxies with different
$n$ in fact have indistinguishable [NUV$-r_{\rm outer}$]–$M_{\ast}$ relations,
thus the differences of [NUV$-r_{\rm central}$]–[NUV$-r_{\rm outer}$] between
different subsamples reflect their different [NUV$-r_{\rm central}$]
distributions. In the upper panels of Figure 3, we show the [NUV$-r_{\rm
central}$]–[NUV$-r_{\rm outer}$] distributions as a function of B/T for the
SFGs with $n$¡2.0. In each mass bin, the SFGs are split into 3 B/T bins. Each
B/T bin includes a similar galaxy number. As shown in Figure 3, the
[NUV$-r_{\rm central}$]–[NUV$-r_{\rm outer}$] distributions are not strongly
dependent on B/T. However, at $\log(M_{\ast}/M_{\sun})<10.8$, SFGs that with
the lowest B/T tend to have bluer [NUV$-r_{\rm central}$]–[NUV$-r_{\rm
outer}$] than the other two B/T bins. In the lower panels of Figure 3, we show
the results for those with $n$¿2.0. Compared to the SFGs with $n$¡2.0, the
[NUV$-r_{\rm central}$]–[NUV$-r_{\rm outer}$] distributions of SFGs with
$n$¿2.0 are broader. As one can see, galaxies with $n$¿2.0 dominate the red
[NUV$-r_{\rm central}$]–[NUV$-r_{\rm outer}$] end. Similarly, the [NUV$-r_{\rm
central}$]–[NUV$-r_{\rm outer}$] distributions are not strongly dependent on
B/T.

In Figure 4, we investigate the dependence of [NUV$-r_{\rm
central}$]–[NUV$-r_{\rm outer}$] on $n$. To minimize the B/T effects, we show
the results in two fixed B/T bins, with B/T=[0.1,0.25] and B/T=[0.25,0.4]. In
both B/T bins, Figure 4 clearly shows that the [NUV$-r_{\rm
central}$]–[NUV$-r_{\rm outer}$] distributions are significantly different for
the $n$¡2.0 and $n$¿2.0 SFGs. Note that the result of the log
($M_{\ast}/M_{\sun}$)=[10.2,10.5] bin may be affected by the relatively small
number of $n$¿2.0 SFGs. Moreover, one can see the $n$¿2.0 SFGs with
B/T=[0.1,0.25] can have a redder [NUV$-r_{\rm central}$]–[NUV$-r_{\rm outer}$]
distribution than that of the $n$¡2.0 SFGs with B/T=[0.25,0.4]. Thus Figure 3
Figure 4 suggest that the star formation of bulges is more strongly dependent
on $n$ than on B/T.

Figure 4.—  The [NUV$-r_{\rm central}$]–[NUV$-r_{\rm outer}$] distributions
for the log($M_{\ast}/M_{\sun}$)¿10.2 SFGs with B/T=[0.1,0.25](upper panels)
and B/T=[0.25,0.4]. From left to right, we show the result for different mass
bins. SFGs with $n$¡2.0 are shown in blue, while those with $n$¿2.0 are in
red. In each bin, the number of the SFGs are marked.

##  5\. Discussion

The NUV$-r$ color is a good proxy of sSFR. However, the color index is usually
affected by metallicity and dust reddening. It is thus necessary to assess the
impacts of metallicity and dust extinction on our results. In Figure 2, we
draw that nearly all galaxies have their NUV$-r_{\rm central}$ and NUV$-r_{\rm
outer}$ bellow 5.0. Over this range, NUV$-r$ is mainly determined by sSFR,
rather than metallicity (Kaviraj et al., 2007a, b). Thus metallicity should
has little influence on our results. On the other hand, if the dust reddening
discrepancy between NUV$-r_{\rm central}$ and NUV$-r_{\rm outer}$ increases
with $M_{\ast}$, then the results of Figure 2 are expected. However, Figure 3
reveals that this should not be the case because at fixed $n$, the peaks of
the [NUV$-r_{\rm central}$]–[NUV$-r_{\rm outer}$] distributions do not shift
redward with increasing $M_{\ast}$. Therefore the increasing color discrepancy
between NUV$-r_{\rm central}$ and NUV$-r_{\rm outer}$ with $M_{\ast}$ shown in
Figure 2 can only be explained by the existence of evolved bulge stellar
population in more massive SFGs.

Figure 3 and Figure 4 support that the color of bulges, or the $\rm sSFR_{\rm
bulge}$ is strongly dependent on $n$. Previous works found that $n$ is very
tightly correlated with quiescence. Above a critical $n$ of $n_{c}$$\sim
2.0-2.5$, galaxies are mostly quenched (Bell et al., 2004; Drory& Fisher,
2007; Bell et al., 2012; Cheung et al., 2012). A high $n_{g}$ also indicates a
dense galaxy bulge. Quenched galaxies mostly host dense cores, however,
galaxies with dense cores do not need to be quenched (Cheung et al., 2012;
Fang et al., 2013). Figure 3 and Figure 4 indicate that the reddest bulges
mostly have $n$¿2.0. However, high $n$ SFGs do not need to have red bulges.
This is consistent with the findings of Cheung et al. (2012) and Fang et al.
(2013).

The dense red bulges of SFGs may imply their formation histories. As already
known, there exists two types of bulges, which are the so-called ”classical
bulges” and ”psuedobulges”. The classical bulges have steep central light
profile (hence dense central stellar density) and are believed to form through
major mergers coupled with a central starburst event, during which the
available fuel is exhaust relatively quickly. Given this, it is thus not
surprised to find that part of the high $n$ SFGs have a red NUV$-r_{\rm
central}$. In contrast, psuedobulges are the bulges that have disk-like
features and believed to be the products of secular evolution (see Kormendy &
Kennicutt (2004) for a review). Figure 3 and Figure 4 show that galaxies with
low $n$ have bluer [NUV$-r_{\rm central}$]–[NUV$-r_{\rm outer}$]
distributions, indicating more SFR in their central regions. This is
consistent with the findings that psuedobulges have more gas and SFR than
classical bulges (Fisher, 2006; Fisher et al., 2013).

At this point it is necessary to discuss the $n$ composition of SFGs. Figure 5
shows the fractions of SFGs with $n$¿2.0 ($f_{\rm n>2.0}$) as a function of
$M_{\ast}$. One can see that the $f_{\rm n>2.0}$ is quite low at
$M_{\ast}<10^{10.2}M_{\sun}$ but rapidly increases to 80%– 90% at
$M_{\ast}\sim 10^{11.0}M_{\sun}$. For comparison, we overplot the $f_{\rm
n>2.0}$ of the SFGs sample from Pan et al. (2015), which is volume-completed
to $M_{\ast}\sim 10^{9.0}M_{\sun}$ at $z<0.05$. One can see that the two SFGs
sample exhibit similar trends. Given this, the MS is expected to turn flatten
owing to the existence of large portion of high $n$ SFGs at the high masses.

Strikingly, [NUV$-r_{\rm central}$]–[NUV$-r_{\rm outer}$] is found to have a
weak dependence on B/T at $M_{\ast}>10^{10.2}M_{\sun}$, regardless of $n$. It
is not straightforward to interpret this result. We argue that this may be due
to the complex mechanisms that drive the increasing of the B/T. The increasing
of B/T can either be due to secular evolution, or due to an intense central
star formation episode. In the former case one sees that high B/T SFGs tend to
have low $\rm SFR_{\rm bulge}$, while in the latter the situation is reversed.
Observationally, both processes can occur in a SFG sample. Even for individual
galaxies, they can possibly undergo both processes during their evolution.
Therefore, for a SFG that with a classical bulge (or a high $n$), the mixture
of the two processes will drive its $\rm SFR_{\rm bulge}$ showing a weak
dependence on B/T.

For the low $n$ SFGs whose central bulges are proposed to mainly form through
secular evolution, we do observed galaxies with lowest B/T have bluest
[NUV$-r_{\rm central}$] at $M_{\ast}<10^{10.8}M_{\sun}$. However, this trend
does not hold when comparing the results between the middle and the high B/T
bins. To interpret this result, one must have a comprehensive picture of the
bulge formation history of massive galaxies. In a recent work, Erwin et al.
(2015) shows evidence that the inner most region of some psuedobulges are
imbedded with a classical bulge, suggesting that our previous proposed bulge
formation pictures are over simple. We suggest a comprehensive understanding
of bulge formation is needed to fully explain the findings presented in this
work.

Figure 5.— The fraction of SFGs that with $n$¿2.0 as a function of $M_{\ast}$.
Data points are calculated in the $M_{\ast}$ bin size of $\Delta M_{\ast}$=0.2
dex. The red symbols are calculated from the SFGs sample used in this work,
while the black symbols are from the SFGs of Pan et al. (2015).

##  6\. Conclusion

In this Letter, we investigate the role of bulges in shaping the
$M_{\ast}$$-$SFR relation by comparting the NUV$-r$ color in the central
$r<R_{50}$ regions to that of the outer regions for a sample of 6401 local
star-forming galaxies. We find that at $M_{\ast}<10^{10.2}M_{\sun}$,
NUV$-r_{\rm central}$ is on average only $\sim$0.25 mag redder than
NUV$-r_{\rm outer}$. Above $M_{\ast}=10^{10.2}M_{\sun}$, [NUV$-r_{\rm
central}$]–[NUV$-r_{\rm outer}$] becomes redder when increasing $M_{\ast}$,
indicating the existence of more evolved bulges at the massive end. For the
galaxies with $M_{\ast}>10^{10.2}M_{\sun}$, We find that those with reddest
[NUV$-r_{\rm central}$]–[NUV$-r_{\rm outer}$] are preferentially to have large
Sérsic index $n$, even at fixed B/T and $M_{\ast}$. This suggests that $\rm
sSFR_{\rm bulge}$ is more strongly dependent on $n$ (or central mass density)
than on B/T. We conclude that the increasing fraction of low-sSFR dense bulges
in $M_{\ast}>10^{10.2}M_{\sun}$ galaxies, rather than high B/T, is responsible
for the flattened slope of the $M_{\ast}$$-$SFR relation at high masses.

This work was supported by the NSFC projects (Grant Nos. 11473053, 11121062,
11233005, U1331201, U1331110, 11225315, 1320101002, 11433005, and 11421303),
the National Key Basic Research Program of China (Grant No. 2015CB857001), the
“Strategic Priority Research Program the Emergence of Cosmological Structures”
of the Chinese Academy of Sciences (Grant No. XDB09000000), the Specialized
Research Fund for the Doctoral Program of Higher Education (SRFDP, No.
20123402110037), and the Chinese National 973 Fundamental Science Programs
(973 program) (2013CB834900, 2015CB857004).

## References

  * Abramson et al. (2014) Abramson K., et al., 2014, ApJ, 785L, 36A
  * Aihara et al. (2011) Aihara, H., Allende Prieto, C., An, D., et al. 2011, ApJS, 193, 29
  * Bell et al. (2004) Bell, E. F., McIntosh, D. H., Barden, M., et al. 2004, ApJ, 600, L11
  * Bell et al. (2012) Bell, E. F., van der Wel, A., Papovich, C., et al. 2012, ApJ, 753, 167
  * Bertin& Arnouts (1996) Bertin, E., & Arnouts, S. 1996, A&AS, 117, 393
  * Brinchmann et al. (2004) Brinchmann, J., Charlot, S., White, S. D. M., et al. 2004, MNRAS, 351, 1151
  * Cheung et al. (2012) Cheung, E, Faber, S., Koo, D., et al. 2012, ApJ, 760, 131
  * Daddi et al. (2007) Daddi, E., Dickinson, M., Morrison, G., et al. 2007, ApJ, 670, 156
  * Drory& Fisher (2007) Drory, N., & Fisher, D. B. 2007, ApJ, 664, 640
  * Elbaz et al. (2007) Elbaz, D., Daddi, E., Le Borgne, D., et al. 2007, A&A, 468, 33
  * Erwin et al. (2015) Erwin, P., Saglia, R. P., Fabricius, M., et al. 2015, MNRAS, 446, 4039
  * Fang et al. (2013) Fang, J. J., Faber, S. M., Koo, D. C., & Dekel, A. 2013, ApJ, 776, 63
  * Fisher& Drory (2008) Fisher, D. B., & Drory, N. 2008, AJ, 136, 773
  * Fisher (2006) Fisher, D. B. 2006, ApJ, 642, L17
  * Fisher et al. (2013) Fisher, D. B., Bolatto, A., Drory, N., et al. 2013, ApJ, 764, 174
  * Fisher& Drory (2011) Fisher, D. B., & Drory, N. 2011, ApJ, 733, L47
  * Guo et al. (2013) Guo, K., Zheng, X. Z., & Fu, H. 2013, ApJ, 778, 23
  * Guo et al. (2015) Guo, K., Zheng, X. Z., Wang, T., & Fu, H. 2015, ApJ, 808, L49
  * Kaviraj et al. (2007a) Kaviraj, S., Rey, S.-C., Rich, R. M., Yoon, S.-J., & Yi, S. K. 2007a, MNRAS, 381, L74
  * Kaviraj et al. (2007b) Kaviraj, S., Schawinski, K., Devriendt, J. E. G., et al. 2007b, ApJS, 173, 619
  * Kauffmann et al. (2003a) Kauffmann G., White S. D. M., Heckman T. M. et al., 2003, MNRAS, 341, 54
  * Kennicutt (1998) Kennicutt, R. C., Jr. 1998, ARA&A, 36, 189
  * Kormendy & Kennicutt (2004) Kormendy, J., & Kennicutt, R. C., Jr. 2004, ARA&A, 42, 603
  * Kroupa (2001) Kroupa, P. 2001, MNRAS, 322, 231
  * Karim et al. (2011) Karim, A., Schinnerer, E., Martínez-Sansigre, A., et al. 2011, ApJ, 730, 61
  * Lee et al. (2015) Lee, N., Sanders, D. B., Casey, C. M., et al. 2015, ApJ, 801, 80
  * Martin et al. (2005) Martin, D. C., Fanson, J., Schiminovich, D., et al. 2005, ApJ, 619, L1
  * Mendel et al. (2014) Mendel, J. T., Simard, L., Palmer, M., Ellison, S. L., & Patton, D. R. 2014, ApJS, 210, 3
  * Pan et al. (2014) Pan, Z., Li, J., Lin, W., Wang, J., & Kong, X. 2014, ApJ, 792, L4
  * Pan et al. (2015) Pan, Z., Li, J., Lin, W., et al. 2015, ApJ, 804, L42
  * Peng et al. (2010) Peng Y. J., et al., 2010, ApJ, 721, 193
  * Pérez et al. (2013) Pérez, E., Cid Fernandes, R., González Delgado, R. M., et al. 2013, ApJ, 764, L1
  * Salim (2014) Salim, S. 2014, Serbian Astronomical Journal, 189, 1
  * Schlegel, Finkbeiner & Davis (1998) Schlegel, D. J., Finkbeiner, D. P., & Davis, M., 1998, ApJ, 500, 525S
  * Schreiber et al. (2015) Schreiber, C., Pannella, M., Elbaz, D., et al. 2015, A&A, 575, A74
  * Simard et al. (2011) Simard, L., Mendel, J. T., Patton, D. R., Ellison, S. L., & McConnachie, A. W. 2011, ApJS, 196, 11
  * Strateva et al. (2001) Strateva, I. V et al., 2001, AJ, 122, 1861
  * Wang et al. (2010) Wang J., et al., 2010, MNRAS, 401, 433
  * Whitaker et al. (2012) Whitaker, K. E., van Dokkum, P. G., Brammer, G., & Franx, M. 2012, ApJ, 754, L29
  * Whitaker et al. (2014) Whitaker, K. E., Franx, M., Leja, J., et al. 2014, ApJ, 795, 104
  * York et al. (2000) York D. G, et al., 2000, AJ, 120, 1579