# MultiModal-Learning for Predicting Molecular Properties: A Framework Based
on Image and Graph Structures

Zhuoyuan Wang1, Jiacong Mi1, Shan Lu2, Jieyue He1,* 1School of Computer
Science and Engineering, Key Lab of Computer Network and Information
Integration, MOE,
Southeast University, Nanjing, 210018, Jiangsu, China
2Nanjing Fenghuo Tiandi Communication Technology Co., Ltd, Nanjing, 211161,
Jiangsu, China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The quest for accurate prediction of drug molecule properties poses a
fundamental challenge in the realm of Artificial Intelligence Drug Discovery
(AIDD). An effective representation of drug molecules emerges as a pivotal
component in this pursuit. Contemporary leading-edge research predominantly
resorts to self-supervised learning (SSL) techniques to extract meaningful
structural representations from large-scale, unlabeled molecular data,
subsequently fine-tuning these representations for an array of downstream
tasks. However, an inherent shortcoming of these studies lies in their
singular reliance on one modality of molecular information, such as molecule
image or SMILES representations, thus neglecting the potential complementarity
of various molecular modalities. In response to this limitation, we propose
MolIG, a novel MultiModaL molecular pre-training framework for predicting
molecular properties based on Image and Graph structures. MolIG model
innovatively leverages the coherence and correlation between molecule graph
and molecule image to execute self-supervised tasks, effectively amalgamating
the strengths of both molecular representation forms. This holistic approach
allows for the capture of pivotal molecular structural characteristics and
high-level semantic information. Upon completion of pre-training, Graph Neural
Network (GNN) Encoder is used for the prediction of downstream tasks. In
comparison to advanced baseline models, MolIG exhibits enhanced performance in
downstream tasks pertaining to molecular property prediction within benchmark
groups such as MoleculeNet Benchmark Group and ADMET Benchmark Group.

###### Index Terms:

Molecular Property Prediction, Contrastive Learning, Molecule Graph, Molecule
Image

##  I Introduction

In the field of computational chemistry and drug discovery, molecular
representation learning is a crucial task aimed at developing effective
methods to represent molecular structures and predict their properties [1, 2,
3]. Accurate prediction of chemical molecule properties can greatly facilitate
the drug development process, reduce research costs, increase the success rate
of new drugs, provide more information and guidance for drug design, and have
significant practical value [4]. Early methods of molecular representation
learning primarily included the topological-based approaches and
physicochemical-based approaches. Topological-based methods describe molecules
by analyzing the chemical bond connections between atoms in molecular
structures, with the Extended Connectivity Fingerprints (ECFP) [5] being one
of the most classic methods. ECFP converts the neighbor information around
atoms into fixed-length binary strings as feature representations. On the
other hand, physicochemical-based methods describe molecular structures by
calculating their physicochemical properties, such as charge states, polarity,
electron affinity, ionization energy and etc. These methods require prior
calculations of physicochemical properties, which are then used as feature
representations. Examples include Molecular Quantum Mechanics (MQM), Molecular
Mechanics (MM), Density Functional Theory (DFT) [6], etc. However, early
methods lacked a deep understanding of molecular structures and the ability to
capture complex molecular features, resulting in certain limitations in
molecular property prediction tasks.

In recent years, the advent of Graph Neural Networks (GNNs) has brought about
remarkable advancements in an array of graph-related tasks [7], subsequently
inspiring their application to the learning of molecular structures. Central
to the concept of a molecular structure-based GNN model is the perception of
the topological structure of atoms and bonds within molecules as a graph,
where atoms and chemical bonds correspond to nodes and edges respectively.
Initial feature sets are formulated based on their inherent physicochemical
properties such as atom type, bond type, and aggregation operations are
executed through the iterative exchange of information amongst neighboring
nodes [8]. In contrast to traditional descriptor-based methods, GNNs can
encapsulate a more extensive set of molecular features, including but not
limited to local interactions and cyclic structures, thereby enhancing the
precision of predictions. To date, numerous GNN-based molecular representation
learning methods have been proposed, such as the Message Passing Neural
Networks (MPNN) [9], and Attentive FP [4], among others.

Simultaneously, graph contrastive learning [10] has been applied to the field
of molecular representation learning with the development of GNNs,
compensating for the scarcity of labeled molecular data and significantly
promoting the development of this field. Existing molecular representation
learning methods based on graph contrastive learning usually adopt a graph
enhancement strategy for molecules. However, data augmentation strategies in
the molecular field are not straightforward due to the specific chemical rules
and constraints of molecular structures, which may necessitate additional
domain knowledge and experience to design suitable molecular augmentation
strategies. In addition to viewing molecules as topological views of nodes and
edges, molecules can also be presented in the form of images [11, 12, 13].
Compared to graph-based representations, molecule image is not dependent on
specific descriptors or encoding methods. Secondly, since molecule image is
two-dimensional, Convolutional Neural Network (CNN) models [14] can be
directly used for analysis. Finally, for some molecules with irregular
structures or high complexity, molecule image may be more suitable for
representation and prediction. These molecules may be difficult to describe
using traditional representation methods, but molecule image can describe the
geometric shape of molecules at a higher level, thereby better capturing the
molecular feature information.

Although previous models based on graphs or SMILES have achieved good
performance, the information provided by a single modality is limited.
Inspired by the significant success of utilizing a large amount of unlabeled
data in Computer Vision (CV) and Natural Language Processing (NLP) , we
introduce the molecule image modality into contrastive learning as an
augmented modality and propose a novel molecular pretraining framework called
MolIG. Considering that molecule graph and molecule image can provide
different levels of chemical and geometric information, we capture the
consistency between them to capture high-order semantic features of molecules.
Compared to previous molecular contrastive learning models [15], our approach
examines molecular features from different perspectives and fully utilizes
information from both modalities. Our contributions can be summarized as
follows:

  * •

We propose a pioneering multimodal pre-training model dedicated to molecular
property prediction, and train the model using both molecular graphs and
molecular images as two modalities.

  * •

Ensuring the preservation of molecular semantic features, we employ three
different image enhancement strategies. By doing so, we maximize the
congruence between molecule graph and image modalities, thereby realizing a
more robust and universally applicable representation learning.

  * •

Our model is subjected to evaluation on both the MoleculeNet Benchmark Group
and ADMET Benchmark Group. The outcomes of the experiment indicate that our
model is proficient not only in extracting the structural attributes of
molecules but also in capturing the more elusive higher-order semantic
information. Furthermore, it surpasses the state-of-the-art methods that rely
on molecule graph and molecule image in terms of results.

##  II Relate Work

###  II-A Graph-Based Molecule Representation

Graph-based molecular representation learning currently stands as the
predominant approach. GROVER[16] ingeniously integrate GNNs and Transformers,
generating node embeddings through the prediction of contextual attributes and
motif information. Simultaneously, contrastive learning has been extensively
employed in the field of molecular representation learning. Conversely, MolCLR
[15] adopted a distinct approach by applying random augmentation operations,
including Node Masking, Edge Deletion, and Subgraph Removal, to the molecule
graph. KANO [17] utilizes knowledge graph to guide molecular graph
enhancement. However, these operations inevitably modify the molecular
structure, thus conflicting with established chemical principles.

###  II-B Image-Based Molecule Representation

ADMET-CNN [11] successfully established a Convolutional Neural Network (CNN)
model predicated on two-dimensional molecule image, yielding excellent
outcomes in forecasting ADMET attributes. Representation learning
methodologies based on molecule image mandate the conversion of data samples
into Euclidean space. Yet, due to the absence of attributes related to atoms
and bonds, direct predictions of properties using molecule image have been
less than satisfactory.

###  II-C MultiModal-Based Molecule Representation

GeomGCL [18] has designed a Geometry Message Passing Network that capitalizes
on both 2D and 3D dual views, adaptively harnessing the vast information
available from both 2D and 3D graphs. MoleculeSTM [19] has harvested an
extensive assortment of descriptive molecular texts from PubChem, forging a
solid bridge of consistency between these narratives and their corresponding
molecule graph. However, to the best of our knowledge, there is currently no
multimodal representation learning method that combines molecular structural
features with image information.

##  III Methods

In this paper, we propose a multimodal molecular pre-training framework
(MolIG) for predicting molecular properties based on Image and Graph
structures, the architecture of which is illustrated in Figure 1. The
framework comprises five components: the Graph Encoder, the Image Encoder, the
Graph Non-Linear Projection, the Image Non-Linear Projection, and the
Contrastive Learning. We will first detail each part of the network
architecture, followed by an explanation of inference on downstream tasks.

###  III-A Graph And Image Encoders

In this section, we will introduce the encoders for the Molecule Graph and the
Molecule Image.

Figure 1: Overview of our method: MolIG.

####  III-A1 Graph Encoder

A molecule graph $G$ can be defined as $G=(V,E)$, where each node $v\in V$,
and each edge $e_{uv}\in E$ signifies the chemical bond between atoms $u$ and
$v$.

$a_{v}^{\left(k\right)}=AGGREGATE^{\left(k\right)}\left(\left\\{h_{u}^{\left(k-1\right)}:u\in\mathcal{N}\left(v\right)\right\\}\right)$
(1)

$h_{v}^{\left(k\right)}=COMBINE^{\left(k\right)}\left(h_{v}^{\left(k-1\right)},a_{v}^{\left(k\right)}\right)$
(2)

$\mathcal{N}\left(v\right)$ represents the set of all neighbors of node $v$,
$h_{v}^{k}$ denotes the representation of atom $v$ at the $k$-th layer. After
$k$ iterations, $h_{v}^{k}$ is able to capture the information of its $k$-hop
neighborhood. The AGGREGATION function collates the information from
neighboring nodes of $v$, and the COMBINE function updates the aggregated
features. The initial representation $h_{v}^{0}$ is initialized by the node
feature $x_{v}$. Notably, in this work, we only employ two types of atomic
attributes, namely atom type and chirality. Similarly, for chemical bonds, we
use bond type and direction as initial features.

To further extract graph-level features $h_{G}$, a readout operation
integrates the features of all nodes in graph $G$, as illustrated in Equation
(3):

$h_{G}=READOUT\left(\left\\{h_{v}^{\left(k\right)}:v\in G\right\\}\right)$ (3)

In this work, we employ the Graph Isomorphism Network (GIN) [20] as the GNN
Encoder. However, as molecule graph differ from other types of graph
structures, edge information significantly impacts the downstream tasks, which
the original GIN does not take into account. In response to this, we follow Hu
et al. [21] to extend node aggregation to
$a_{v}^{\left(k\right)}=\sum_{u\in\mathcal{N}\left(v\right)}\sigma\left(h_{u}^{\left(k-1\right)}+e_{uv}\right)$,
which considers both node and edge information simultaneously. Here,
$\sigma$(·) is a non-linear activation function. The readout operation is an
average pooling function, used to obtain the graph-level representation for
each molecule. The GIN network has 5 million parameters.

####  III-A2 Image Encoder

For molecule image, we adopt ResNet [14] as the encoder to extract features
from each molecule image. In our task, due to the lightweight nature of
ResNet-34, we ultimately chose it as the image backbone. It is worth noting
that there is a significant discrepancy between molecule image and ImageNet
[22], so we do not use ResNet pre-trained on ImageNet. The Resnet network has
22 million parameters. Instead, we trained the image encoder from scratch to
accommodate the unique properties of molecule image, a decision that we
validated during our experimentation.

###  III-B Graph and Image Projection Heads

The projection head serves as a pivotal component in the MolIG model, mapping
high-dimensional feature vectors extracted from the Graph Encoder and Image
Encoder into a lower-dimensional space. Specifically, the projection head can
be regarded as a Multilayer Perceptron (MLP), which aims to learn more
discriminative representations in the lower-dimensional space for contrastive
learning. We follow the nonlinear MLP mentioned in [23], adding a Relu
activation function in the hidden layer. Given the substantial disparity
between molecule graph and molecule image, we deploy two identical, albeit
non-shared-parameter, projection heads. This approach facilitates superior
results in downstream tasks of molecular property prediction. The projection
head can be formalized as follows:

$z_{i}^{graph}=PrejectionHead_{graph}\left(h_{i}^{graph}\right)$ (4)

$z_{i}^{image}=PrejectionHead_{image}\left(h_{i}^{image}\right)$ (5)

Where $PrejectionHead_{graph}$ and $PrejectionHead_{image}$ denote the Non-
Linear Projection Heads of the GNN Encoder and Image Encoder, respectively.

###  III-C Contrastive Learning Framework

Specifically, given a batch of $N$ molecules $\\{M_{0},M_{1}\ldots
M_{n-1}\\}$, we first convert them into Graphs $\\{G_{0},G_{1},G_{2}\ldots
G_{n-1}\\}$ and Images $\\{I_{0},I_{1},I_{2}\ldots I_{n-1}\\}$ via Rdkit [24].
The molecule image resolution is 224x244. Graphs and Images derived from the
same molecule constitute positive samples, whereas graphs from different
molecules are considered negative samples. The GNN-based Graph Encoder encodes
$G_{i}$ into $h_{i}^{graph}$, and the ResNet-based Image Encoder encodes
$I_{i}$ into $h_{i}^{image}$. However, due to significant differences between
the modalities, directly using $h_{i}^{graph}$ and $h_{i}^{image}$ as inputs
for contrastive loss often results in suboptimal performance in downstream
tasks. To address this issue, we design separate mapping heads for the graph
and image modalities, respectively mapping the molecule graph and molecule
image to the same projection space to compute the contrastive loss, that is,
$z_{i}^{graph}$ and $z_{i}^{image}$.

Our goal is to maximize the representation similarity of positive sample
molecules under Graph and Image modalities. Therefore, under the Graph-Image
view, following SimCLR [23], we use the normalized temperature-scaled cross
entropy (NT-Xent) as the loss function:

$\mathcal{L}_{{i},{j}}=-log\frac{exp{\left(sim\left(z_{i}^{graph},z_{j}^{image}\right)/\tau\right)}}{\sum_{k=1}^{2N}{1({k}\neq
i)}exp{\left(sim\left(z_{i}^{graph},z_{k}^{image}\right)/\tau\right)}}$ (6)

Where $\tau$ denotes the temperature coefficient, and we implement the
calculation of similarity as
$sim(z_{i},z_{j})=\frac{z_{i}^{T}z_{j}}{\left.||z_{i}\right.||_{2}\textperiodcentered\left.||z_{j}\right.||_{2}}$.
The final loss is calculated within a mini-batch.

###  III-D Fine-tuning and Downstream Inference

In this study, we adopt a strategy of pre-training plus fine-tuning to
complete the task of drug property prediction. We discard the Image Encoder
from the pre-training phase, retaining only the Graph Encoder as the drug
molecule encoder, using the pre-trained model to learn the molecular
representation of the drug. Subsequently, we fine-tune the pre-trained model
on the task of drug property prediction, enabling it to better adapt to the
data distribution of the specific task. Specifically, we add only an
additional fully connected layer, PredictionHead($w$), after the Graph
Encoder, where $w$ represents the parameters of the fully connected layer. In
the fine-tuning process, we first input the molecule graph of the drug,
$Drug_{i}$, into the GNN to obtain the latent representation of the molecule,
$h_{i}$, and then forward it to the PredictionHead to make the final property
prediction, formalized as follows:

$\hat{y}=PredictionHead\left(GNN\left(Drug_{i}\right)\right)$ (7)

##  IV Experiments

###  IV-A Pretraining Dataset and Data Augmentations

For the pre-training dataset, we randomly select 10 million unlabeled drug-
like molecules from the 114 million molecules in the Pubchem database [25]. We
then divide the pre-training dataset into training and validation sets at a
ratio of 0.95/0.05.

Data augmentation serves as an effective approach to enhance the
generalization capability and robustness of a model. By introducing more data
variations during the training process, models can better adapt to the complex
scenarios encountered in the real world. Data augmentation has been widely
applied in computer vision and multimodal domains. However, for molecule
graph, common augmentation techniques such as random atom masking, edge
perturbation, or subgraph sampling can disrupt molecular structural
information, resulting in learned representations that do not fit well with
the data in downstream tasks. Therefore, we do not perform any augmentation
operations on molecule graph during the pre-training phase. In contrast,
molecule image is sparser compared to real-world images, with over 90% of
their regions filled with zeros, meaning that only a minimal fraction of the
pixels are truly relevant to downstream tasks [13]. In light of this, we chose
three augmentation strategies during the pre-training phase: (1)
RandomHorizontalFlip, (2) Ran-domGrayscale, and (3) RandomRotation. Each
strategy has a 25% probability of being executed. These strategies do not
alter the structure of the molecule image and enable the model to learn the
invariance introduced by data augmentation.

TABLE I: Test performance of different models on six classification
benchmarks. The first two models are supervised learning methods and the last
eight are self-supervised/pre-training methods. Mean and standard deviation of
test ROC-AUC (%) on each benchmark are reported.

Dataset BBBP Tox21 ClinTox HIV BACE SIDER AVG #molecule 2039 7831 1478 41127
1513 1478 - #task 1 12 2 1 1 27 - Split Scaffold Scaffold Scaffold Scaffold
Scaffold Scaffold - GCN 71.8$\pm$0.1 70.9$\pm$2.6 62.5$\pm$2.8 74.0$\pm$3.0
71.6$\pm$2.0 53.6$\pm$3.2 67.4 GIN 65.8$\pm$4.5 74.0$\pm$0.8 58.0$\pm$4.4
75.3$\pm$1.9 70.1$\pm$5.4 57.3$\pm$1.6 66.8 GraphCL 67.5$\pm$3.3 74.4$\pm$0.7
78.9$\pm$4.2 75.0$\pm$0.4 68.7$\pm$7.8 60.1$\pm$1.3 70.8 GROVER 68.0$\pm$1.5
76.3$\pm$0.6 76.9$\pm$1.9 77.8$\pm$1.4 79.5$\pm$1.1 60.7$\pm$0.5 73.2 MGSSL
69.7$\pm$0.9 76.5$\pm$0.3 80.7$\pm$2.1 78.8$\pm$1.2 79.1$\pm$0.9 61.8$\pm$0.8
74.4 3DInfoMax 69.1$\pm$1.0 74.5$\pm$0.7 59.4$\pm$3.2 76.1$\pm$1.3
79.4$\pm$1.9 53.3$\pm$3.3 68.7 GraphMVP 72.4$\pm$1.6 74.4$\pm$0.2 77.5$\pm$4.2
77.0$\pm$1.2 81.2$\pm$0.9 63.9$\pm$1.2 74.4 GraphMAE 72.0$\pm$0.6 75.5$\pm$0.6
82.3$\pm$1.2 77.2$\pm$1.0 83.1$\pm$0.9 60.3$\pm$1.1 75.1 MolCLR 71.6$\pm$0.7
75.0$\pm$0.4 91.9$\pm$1.2 78.3$\pm$0.4 81.9$\pm$1.5 59.9$\pm$0.9 76.4 MolIG
73.4$\pm$0.7 75.9$\pm$0.0 89.1$\pm$2.0 79.8$\pm$0.6 84.5$\pm$0.2 66.1$\pm$0.5
78.1

###  IV-B Molecule Property Prediction on MoleculeNet

####  IV-B1 Dataset

To evaluate the performance of MolIG, we fine-tune the pre-trained model on
six classification datasets from MoleculeNet [2], including BBBP, Tox21,
ClinTox, HIV, BACE, and SIDER. Compared to random split, scaffold split is a
more challenging and realistic method of division, greatly testing the
robustness and generalization capabilities of the model. We use a scaffold
split to divide the above six datasets into training/validation/test sets at a
ratio of 0.8/0.1/0.1.

####  IV-B2 Baseline

We compare MolIG with state-of-the-art baseline models for molecular property
prediction, which includes supervised learning models like GCN [26] and GIN
[20]. The rest of the models are pre-trained models.

  * •

GraphCL [10] learns molecular representations by maximizing the mutual
information of graphs under two different augmentation strategies.

  * •

3DInfoMax[27] and GraphMVP[28] introduce 3D molecular information and maximize
the representation of molecules in both 2D and 3D.

  * •

GROVER [16] combines GNN and Transformer, learning molecular representations
through two pre-training tasks, namely context prediction and Motif
prediction.

  * •

MGSSL [29] proposes a universal Motif-based generative pre-training framework,
which requires GNN to perform topology and label prediction.

  * •

GraphMAE [30] applies MAE [31] to Graph, learning effective information by
reconstructing node features.

  * •

MolCLR [15] performs contrastive learning with molecule graph under two
identical augmentation strategies.

####  IV-B3 Result

In Table 1, we report the average ROC-AUC values and standard deviations of
MolIG on six benchmark datasets. We independently ran MolIG 3 times on each
dataset. Notably, MolIG achieve the best performance on five of the datasets
and reached the highest average value across all datasets. We attribute the
exceptional performance exhibited by MolIG to the effectiveness of the pre-
training model in integrating molecule image and graph structure information.
These two modalities together provide a rich context for molecular property
prediction, thereby improving the accuracy of the predictions. Specifically,
compared to the SOTA model MolCLR, which only uses a single molecule graph
modality, MolIG has a significant performance advantage, surpassing MolCLR in
five of the datasets. Particularly on the SIDER dataset, MolIG achieve a
performance improvement of 6.2%. Especially on the ClinTox dataset, MolIG
achieve a performance improvement of 4.0%. This demonstrates the superiority
of the multimodal strategy adopted by MolIG in excavating the latent knowledge
in molecular data.

###  IV-C Molecule Property Prediction on ADMET

####  IV-C1 Dataset

ADMET [32] is a crucial concept in drug discovery, representing properties
such as drug absorption, distribution, metabolism, excretion, and toxicity.
Similar to the MoleculeNet Benchmark Group, we fine-tune MolIG on different
tasks in the ADMET Benchmark Group. But with the difference that, we use the
molecule scaffold division provided by ADMET to divide each dataset into
training, validation, and testing sets in a 0.8/0.1/0.1 ratio.

####  IV-C2 Baseline

In this study, we achieve MolIG with three benchmark models: GCN [26],
Attentive FP [4], and MolCLR [15].

  * •

GCN [26] and MolCLR [15] are consistent with the models we used in the
MoleculeNet Benchmark Group.

  * •

Attentive FP[4] combines the concepts of attention mechanism and molecular
fingerprints to capture complex interactions between atoms and bonds in a
molecule. It iteratively updates the features of atoms and bonds to generate
vector representations of the molecules.

TABLE II: Test performance of different models on ADMET benchmark group. Mean result on each benchmark are reported. Category | Dataset | Metric | GCN | AttentiveFP | MolCLR | MolIG
---|---|---|---|---|---|---
Absorption | Caco2 | MAE | 0.599 | 0.401 | 0.434 | 0.341
HIA | ROC-AUC | 0.936 | 0.974 | 0.956 | 0.966
Pgp | ROC-AUC | 0.895 | 0.892 | 0.861 | 0.921
Bioavailability | ROC-AUC | 0.566 | 0.632 | 0.738 | 0.725
Lipophilicity | MAE | 0.541 | 0.572 | 0.496 | 0.461
Solubility | MAE | 0.907 | 0.776 | 0.776 | 0.753
Distribution | PPBR | MAE | 10.194 | 9.373 | 9.196 | 8.499
VDss | Spearman | 0.457 | 0.241 | 0.547 | 0.717
Metabolism | CYP2D6 inhibition | PR-AUC | 0.616 | 0.646 | 0.686 | 0.693
CYP3A4 inhibition | PR-AUC | 0.840 | 0.851 | 0.840 | 0.841
CYP2C9 inhibition | PR-AUC | 0.735 | 0.749 | 0.781 | 0.802
CYP2D6 substrate | PR-AUC | 0.617 | 0.547 | 0.486 | 0.702
CYP3A4 substrate | ROC-AUC | 0.590 | 0.576 | 0.543 | 0.655
CYP2C9 substrate | PR-AUC | 0.344 | 0.375 | 0.353 | 0.400
Excretion | Half life | Spearman | 0.239 | 0.085 | 0.133 | 0.357
Clearance microsome | Spearman | 0.532 | 0.365 | 0.525 | 0.561
Clearance hepatocyte | Spearman | 0.366 | 0.289 | 0.386 | 0.387
Toxicity | hERG | ROC-AUC | 0.738 | 0.825 | 0.821 | 0.831
Ames | ROC-AUC | 0.818 | 0.814 | 0.887 | 0.883
DILI | ROC-AUC | 0.856 | 0.886 | 0.871 | 0.921
LD50 | MAE | 0.649 | 0.678 | 0.435 | 0.428

####  IV-C3 Result

In Table 2, we report the average ROC-AUC values for MolIG on 21 benchmark
datasets in ADMET. Compared to the MoleculeNet Benchmark Group, the ADMET
Benchmark Group focuses more on the behavioral characteristics of drug
molecules within biological entities, and therefore, contains relatively more
complex molecular structures. The experimental results show that MolIG
achieves the best performance on 17 out of the 21 datasets. Except for the
Bioavailability and Ames datasets, our model outperformed MolCLR, which only
uses Graph modality. Particularly, in the Pgp dataset, we improve the
performance by nearly 3% compared to the best baseline. In the VDss dataset,
the performance was improved by nearly 17%. In the CYP2D6 substrate, MolIG
achieves a performance improvement of nearly 9%. These results suggest that by
integrating molecule image information into the model, MolIG has a more
significant advantage in predicting more complex properties within biological
entities. The outstanding performance of MolIG on the ADMET benchmark datasets
confirms the effectiveness of the multimodal pre-training framework in the
field of molecular property prediction. This research provides useful insights
for further exploring multimodal learning strategies to improve the accuracy
of drug molecule property prediction in the future.

###  IV-D Ablation Study

This ablation study aims to evaluate whether pre-training strategies and data
augmentation strategies can help the model achieve better performance in
downstream tasks. As shown in Figure 2, MolIG, which utilizes both pre-
training and data augmentation strategies (represented by the grey bar),
performs best among all model architectures. Models without pre-training
(represented by ’w/o pretrain’) perform worst in all tasks. Pre-training
strategies without data augmentation (represented by ’w/o img aug’) perform
second best, yet they show a significant performance increase compared to
models without any pre-training. Excluding either of these two components can
easily lead to a decrease in performance.

Compared to models completely without pre-training, MolIG improves the ROC-AUC
metric by 13.0% on the ClinTox dataset, by 7.8% on the SIDER dataset, and an
average of 6.5% across all six datasets. The ’w/o img aug’ strategy, although
not using data augmentation, still shows a noticeable improvement, indicating
the effectiveness of our pre-training strategy. MolIG improves across all six
datasets, suggesting that the data augmentation strategy further enhances
robustness and generalization capabilities of the model.

In conclusion, multimodal pre-training strategy of MolIG can learn high-level
semantic information from a large amount of unlabeled molecular data, enabling
the model to learn more discriminative representations. Meanwhile, the data
augmentation strategy enhances robustness and generalization capabilities by
perturbing the image modality without changing the semantic information and
effectively transfers to relevant biological tasks for molecular property
prediction.

Figure 2: Ablation Study: Impact of Pre-training and Data Augmentation
Strategies on the Performance of MolIG on Six Classification Datasets in
MoleculeNet.

###  IV-E Parameter Experiment

A critical hyperparameter in the MolIG model is the temperature scaling used
in contrastive learning, which adjusts the sensitivity of the model to the
similarity scores of positive and negative samples. As shown in Table 3, to
investigate the influence of different temperature coefficients on the
performance of downstream tasks, we adopt three different temperature
coefficients during the pre-training phase of the model, namely 0.05, 0.1, and
0.5, and conduct experiments on six classification datasets on MoleculeNet,
reporting the average ROC-AUC values under these three temperature
coefficients. Moreover, Figure 3 illustrates the specific performance of
different temperature coefficients on these datasets.

From Table 3 and Figure 3, it can be observed that when the temperature
coefficient is 0.1, the performance of the model on downstream tasks reaches
its peak. We posit that a lower temperature coefficient assists the model in
distinguishing negative samples more effectively. However, when the
temperature coefficient is 0.05, its average performance on downstream tasks
is 2% lower than that of the model with a temperature coefficient of 0.1. This
may be due to the sparsity of molecule image causing the model to transmit
redundant information from the image modality to the GNN encoder. A higher
temperature coefficient, on the other hand, simplifies the model training
process, preventing it from learning meaningful representations. We have
observed that when the temperature coefficient is set to 0.5, the model
struggles to converge, leading to unstable training. Moreover, it reduces the
discriminative ability between different molecules, resulting in ambiguous
predictions. This could significantly impact its performance in downstream
tasks.

TABLE III: The Impact of Temperature ($\tau$) on the Loss of MolIG: A Report on the Average Performance across Six Classification Datasets. Temperature ($\tau$) | 0.05 | 0.1 | 0.5 |
---|---|---|---|---
ROC-AUC (%) | 76.1 | 78.1 | 75.0 |
Figure 3: Performance of MolIG on six classification datasets in MoleculeNet
under different temperature coefficients.

###  IV-F Molecule Retrieval via MolIG

In order to further explore the expressive power of MolIG, we employ MolIG in
a molecular similarity search experiment to further assess its performance. In
the experiment, we randomly select 100,000 molecules from the PubChem database
as query objects and provided a SMILES string for each molecule. For each
given Query molecule, we first use MolIG to obtain its molecular
representation and then calculated the cosine similarity with the
representations of other molecules in the database. Based on the similarity
scores, the top 9 ranked molecules are displayed. We select two instances: a
molecule with a unique substructure,
CC[S@SP3](=O)(=O)c1cccnc1c1nnc(c2ccc(C)cn2)o1, and a molecule with a heavy
element, CCn1ncc(Br)c1C(=O)CCC(C)
(C)N. As can be seen from Figure 4, the queried molecules not only capture the
multi-ring structure in the Query molecule but also capture its S=O functional
group. Similarly, the molecules queried in Figure 5 almost all capture the
element Br in the Query molecule. Even if Br is not present, they can still
capture elements like F or Cl, as they belong to the same halogen group in the
periodic table and share similar chemical properties. These experimental
results further demonstrate that MolIG can not only extract structural
features of molecules but also capture higher-order semantic information. Even
at a finer granularity level, our model still possesses a strong
representational capability.

Figure 4: Query results for the molecule CC[S@SP3](=O)(=O)c1cccnc1c1
nnc(c2ccc(C)cn2)o1. Figure 5: Query results for the molecule
CCn1ncc(Br)c1C(=O)CCC(C)(C)N.

##  V Implemention Ddtails

Our pre-training model consists of a Graph Isomorphism Network (GIN) with 5
layers and 300 hidden dimensions and a residual convolutional neural network
(ResNet-34) with 22M parameters. We pre-train the model for 50 epochs using a
batch size of 512 on 1 RTX 3090 GPU. We use the Adam optimizer, the ResNet
learning rate is 0.01 and the GIN learning rate is 0.0005. The temperature
coefficient for Contrastive loss is 0.01. The weight decay is 1e-5. We take
image with resolution of 224×224.

##  VI Conclusions

Molecular property prediction plays a crucial role in drug discovery. While
previous models for predicting molecular properties have achieved considerable
success, the use of a single modality of information often limits their
predictive performance. In this study, we propose a novel multi-modal
molecular pre-training framework MolIG which learns molecular representations
by maximizing the consistency between the features of the Graph and Image
modalities during the pre-training phase. The empirical evaluations on drug
discovery tasks such as MoleculeNet and ADMET demonstrate that MolIG
outperforms the current state-of-the-art baseline models. Notably, MolIG not
only extracts structural features of molecules but also captures higher-order
semantic information, which can be transferred to biologically relevant tasks
in molecular property prediction. Furthermore, our model currently only
considers information from two modalities. Incorporating SMILES and 3D
information into the framework will be the subject of our future research.

## Acknowledgments

This work was supported by National Key R&D Program of China (2019YFC1711000)
and Collaborative Innovation Center of Novel Software Technology and
Industrialization.

## References

  * [1] Garrett B Goh, Nathan O Hodas, and Abhinav Vishnu.  Deep learning for computational chemistry.  Journal of computational chemistry, 38(16):1291–1307, 2017.
  * [2] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande.  Moleculenet: a benchmark for molecular machine learning.  Chemical science, 9(2):513–530, 2018.
  * [3] Hongming Chen, Ola Engkvist, Yinhai Wang, Marcus Olivecrona, and Thomas Blaschke.  The rise of deep learning in drug discovery.  Drug discovery today, 23(6):1241–1250, 2018.
  * [4] Zhaoping Xiong, Dingyan Wang, Xiaohong Liu, Feisheng Zhong, Xiaozhe Wan, Xutong Li, Zhaojun Li, Xiaomin Luo, Kaixian Chen, Hualiang Jiang, et al.  Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism.  Journal of medicinal chemistry, 63(16):8749–8760, 2019.
  * [5] David Rogers and Mathew Hahn.  Extended-connectivity fingerprints.  Journal of chemical information and modeling, 50(5):742–754, 2010\.
  * [6] Maylis Orio, Dimitrios A Pantazis, and Frank Neese.  Density functional theory.  Photosynthesis research, 102:443–453, 2009.
  * [7] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip.  A comprehensive survey on graph neural networks.  IEEE transactions on neural networks and learning systems, 32(1):4–24, 2020.
  * [8] Zhichun Guo, Bozhao Nan, Yijun Tian, Olaf Wiest, Chuxu Zhang, and Nitesh V Chawla.  Graph-based molecular representation learning.  arXiv preprint arXiv:2207.04869, 2022.
  * [9] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl.  Neural message passing for quantum chemistry.  In International conference on machine learning, pages 1263–1272. PMLR, 2017.
  * [10] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen.  Graph contrastive learning with augmentations.  Advances in neural information processing systems, 33:5812–5823, 2020.
  * [11] Tingting Shi, Yingwu Yang, Shuheng Huang, Linxin Chen, Zuyin Kuang, Yu Heng, and Hu Mei.  Molecular image-based convolutional neural network for the prediction of admet properties.  Chemometrics and Intelligent Laboratory Systems, 194:103853, 2019\.
  * [12] Shifa Zhong, Jiajie Hu, Xiong Yu, and Huichun Zhang.  Molecular image-convolutional neural network (cnn) assisted qsar models for predicting contaminant reactivity toward oh radicals: Transfer learning, data augmentation and model interpretation.  Chemical Engineering Journal, 408:127998, 2021.
  * [13] Garrett B Goh, Charles Siegel, Abhinav Vishnu, Nathan O Hodas, and Nathan Baker.  Chemception: a deep neural network with minimal chemistry knowledge matches the performance of expert-developed qsar/qspr models.  arXiv preprint arXiv:1706.06689, 2017.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [15] Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani.  Molecular contrastive learning of representations via graph neural networks.  Nature Machine Intelligence, 4(3):279–287, 2022.
  * [16] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang.  Self-supervised graph transformer on large-scale molecular data.  Advances in Neural Information Processing Systems, 33:12559–12571, 2020.
  * [17] Yin Fang, Qiang Zhang, Ningyu Zhang, Zhuo Chen, Xiang Zhuang, Xin Shao, Xiaohui Fan, and Huajun Chen.  Knowledge graph-enhanced molecular contrastive learning with functional prompt.  Nature Machine Intelligence, pages 1–12, 2023.
  * [18] Shuangli Li, Jingbo Zhou, Tong Xu, Dejing Dou, and Hui Xiong.  Geomgcl: Geometric graph contrastive learning for molecular property prediction.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4541–4549, 2022.
  * [19] Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang, Chaowei Xiao, and Anima Anandkumar.  Multi-modal molecule structure-text model for text-based retrieval and editing.  arXiv preprint arXiv:2212.10789, 2022.
  * [20] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka.  How powerful are graph neural networks?  In International Conference on Learning Representations, 2019.
  * [21] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec.  Strategies for pre-training graph neural networks.  In International Conference on Learning Representations, 2020.
  * [22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, 2017.
  * [23] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [24] Greg Landrum.  Rdkit: Open-source cheminformatics. 2006.  Google Scholar, 2006.
  * [25] Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A Shoemaker, Paul A Thiessen, Bo Yu, et al.  Pubchem 2019 update: improved access to chemical data.  Nucleic acids research, 47(D1):D1102–D1109, 2019.
  * [26] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In International Conference on Learning Representations (ICLR), 2017\.
  * [27] Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Liò.  3d infomax improves gnns for molecular property prediction.  In International Conference on Machine Learning, pages 20479–20502. PMLR, 2022.
  * [28] Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang.  Pre-training molecular graph representation with 3d geometry.  In International Conference on Learning Representations, 2022.
  * [29] Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee.  Motif-based graph self-supervised learning for molecular property prediction.  Advances in Neural Information Processing Systems, 34:15870–15882, 2021.
  * [30] Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang.  Graphmae: Self-supervised masked graph autoencoders.  In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 594–604, 2022.
  * [31] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022.
  * [32] Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik.  Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development.  arXiv preprint arXiv:2102.09548, 2021.


# Crystal Toolkit: A Web App Framework to Improve Usability and Accessibility
of Materials Science Research Algorithms

Matthew Horton<EMAIL_ADDRESS>Materials Science Division, Lawrence Berkeley
National Lab  Department of Materials Science and Engineering, University of
California, Berkeley  Jimmy-Xuan Shen  Materials Science Division, Lawrence
Berkeley National Lab  Department of Materials Science and Engineering,
University of California, Berkeley  Jordan Burns  Department of Materials
Science and Engineering, University of California, Berkeley  Energy Storage
and Distributed Resources Division, Lawrence Berkeley National Lab  Orion
Cohen  Materials Science Division, Lawrence Berkeley National Lab  Department
of Chemistry, University of California, Berkeley  François Chabbey  Materials
Science Division, Lawrence Berkeley National Lab  Alex M. Ganose  Department
of Chemistry, Imperial College London  Rishabh Guha  Materials Science
Division, Lawrence Berkeley National Lab  Patrick Huck  Materials Science
Division, Lawrence Berkeley National Lab  Haoming Howard Li  Department of
Materials Science and Engineering, University of California, Berkeley
Materials Science Division, Lawrence Berkeley National Lab  Matthew McDermott
Department of Materials Science and Engineering, University of California,
Berkeley  Materials Science Division, Lawrence Berkeley National Lab  Joseph
Montoya  Toyota Research Institute  Guy Moore  Department of Materials Science
and Engineering, University of California, Berkeley  Materials Science
Division, Lawrence Berkeley National Lab  Jason Munro  Materials Science
Division, Lawrence Berkeley National Lab  Cody O’Donnell  Materials Science
Division, Lawrence Berkeley National Lab  Colin Ophus  The National Center for
Electron Microscopy, Molecular Foundry, Lawrence Berkeley National Lab  Guido
Petretto  Matgenix SRL  Modelling Division, Institute of Condensed Matter and
Nanosciences, Université catholique de Louvain  Janosh Riebesell  Cavendish
Laboratory, University of Cambridge  Molecular Foundry, Lawrence Berkeley
National Lab  Steven Weitzner  Lawrence Livermore National Laboratory  Brook
Wander  Department of Chemical Engineering, Carnegie Mellon University  Donald
Winston  Energy Technologies Area, Lawrence Berkeley National Lab  Ruoxi Yang
Materials Science Division, Lawrence Berkeley National Lab  Steven Zeltmann
Department of Materials Science and Engineering, University of California,
Berkeley  Anubhav Jain  Materials Science Division, Lawrence Berkeley National
Lab  Kristin A. Persson<EMAIL_ADDRESS>Molecular Foundry, Lawrence
Berkeley National Lab  Department of Materials Science and Engineering,
University of California, Berkeley

###### Abstract

Crystal Toolkit is an open source tool for viewing, analyzing and transforming
crystal structures, molecules and other common forms of materials science data
in an interactive way. It is intended to help beginners rapidly develop web-
based apps to explore their own data or to help developers make their research
algorithms accessible to a broader audience of scientists who might not have
any training in computer programming and who would benefit from graphical
interfaces. Crystal Toolkit comes with a library of ready-made components that
can be assembled to make complex web apps: simulation of powder and single
crystalline diffraction patterns, convex hull phase diagrams, Pourbaix
diagrams, electronic band structures, analysis of local chemical environments
and symmetry, and more. Crystal Toolkit is now powering the Materials Project
website frontend, providing user-friendly access to its database of computed
materials properties. In the future, it is hoped that new visualizations might
be prototyped using Crystal Toolkit to help explore new forms of data being
generated by the materials science community, and that this in turn can help
new materials scientists develop intuition for how their data behaves and the
insights that might be found within. Crystal Toolkit will remain a work-in-
progress and is open to contributions from the community.

##  I Introduction

Effective communication of scientific results is becoming ever more essential:
an exponential increase in quantity of papers produced has lead to an
“attention decay”[1], whereby it is becoming increasingly difficult for
researchers to stay abreast of the published literature. Furthermore, new
scientific techniques are generating a vast increase in the quantity of data:
this, too, can present a problem, as insights are hidden in these large data
sets, which are often difficult to explore. This is a challenge to our
community and also an opportunity to find new solutions for better scientific
communication. This paper describes improvement in this area by the
introduction of a new tool for computational materials scientists to help them
explore and share their data, and increase the impact of their work by making
it easier to understand and build connections between data sets.

Materials science and chemistry have always benefited from a visual
exploration of concepts, whereby visual representations have not only been
used as a tool to accelerate learning but have actually enabled new scientific
discoveries[2] (see Figure 1). These include Dalton’s atoms at the advent of
atomic theory, Kekulé’s affinity units (initially a convenient orthographic
device but that led to the discovery of the benzene ring), Hofmann’s molecules
built from croquet balls (introducing a strong sense of the spatial
arrangement of molecules, and colour schemes for elements that are still in
use today), van’t Hoff’s asymmetric carbon and the concept of stereoisomers,
Barlow and the understanding of close packing in crystal structures, Hodgkin
and the visualization of electron charge densities in real-space, and even
Megaw’s popularization of materials science through the Festival Pattern
Group–the list goes on! These examples demonstrate how effective visualization
has led to essential leaps throughout the history of materials science, and
justifies close attention to this area – for education[3, 4] as well as
research.

In the modern era, there has been an explosion of options for molecular
visualizations enabled by advances in computation which support new
computational simulation techniques. A full review of these would be a
substantial effort, but include XCrysDen[5], OVITO[6], VESTA[7], Avogadro[8],
VMD[9], Jmol[10], 3DMol[11], nglviewer[12], AtomEye[13]. Each of these tools
has developed and refined what is possible in molecular visualization, and
have, in turn, each contributed to scientific insights. It is reasonable
therefore to consider that the problem of molecular visualization might be
solved and ask what benefit yet another option might bring.

Crystal Toolkit has been developed to address new, specific requirements
emerging from the computational materials science community. It provides not
only new means of molecular and crystal visualization, but goes beyond these
and facilitates the visualization of additional forms of data relevant to the
field. These requirements come specifically as a result of the rise of high-
throughput computational approaches and the vast quantities of data that they
create, as well as from recent efforts which closely link computation with
experiment to aid in the design of new materials.

Figure 1: A timeline of molecular and crystallographic visualizations with
several key advances highlighted. Throughout the history of chemistry and
materials science, effective visualization has proven an invaluable tool to
help develop intuition and unlock key new insights that have advanced the
field.

##  II Design Goals

Several key goals have governed the design of Crystal Toolkit, informed by
historical and technological context, as well as the current needs of the
computational materials science community. These include:

  * •

A human-centered approach. Since the early study of “human-computer
interaction”[14], people have framed the relationship between human and
computer as a dialogue. Unfortunately, this dialogue is often too focused on
the computer, and humans are forced to adapt to the machine rather than the
inverse. In scientific research, this often manifests as poorly-documented
codes which are difficult to use or compile, and has had a tremendous cost on
scientific advance: in the best case, it delays progress of new users by
months or years as they have to learn the quirks of these computational tools,
while in the worst case it can alienate new scientists and drive them out of
the field entirely, or even result in the publication of incorrect results due
to misunderstanding and misinterpretation of a computer code or its output. It
is critical that the next generation of tools that we develop as a community
start with a human-centered approach to avoid these outcomes.

  * •

Strong mapping of visualizations to specific software objects. Individual
software objects (for example, a Structure object representing a crystal
structure) are often difficult to introspect and understand what information
they contain. If you are a proficient programmer you can inspect its
attributes or read the relevant source code, but this is often inaccessible to
new learners. Visualizations allow both novice and experienced users to
leverage the brain’s immense capacity for visual information processing[15].
Visual representations of data structures might include a formal
representation such as a UML (Unified Modeling Language) diagram, but might
more usefully be a 3D visualization or a graphical plot. Crystal Toolkit aims
to provide understandable visualizations for the commonly used software
objects used by the materials science community.

  * •

Faithful representations of software objects. It is important that a
visualization faithfully represents an underlying software object, and that
this in turn is a faithful representation of the corresponding scientific
abstraction (see Figure 2). These two degrees of separation–between concept
and code, and code and visualization–are fraught, and can introduce problems.
For a specific example, some molecular visualizations will automatically
detect and insert bonds that are not present in the underlying software
object. While useful, it impedes understanding of the object, and can
introduce confusion for new developers. A better option is to only visualize
the information actually present in the object by default with the option of
including additional ornaments.

  * •

Composability of visualizations. Object-orientated programming is a common
pattern in modern programming, whereby complex objects can inherit from and be
composed from simpler objects (see Figure 3). Where possible, if a complex
object is built from a set of simpler objects, and these objects in turn
already have visualizations available, then it should be possible that the
visualization of the more complex object can be composed from these already
existing visualizations. This allows for reuse and experimentation, and helps
enforce the faithfulness of the representation.

  * •

Empowering users of all experience levels. Many scientists do not receive
formal training in programming, or might receive training in an informal and
ad-hoc manner. Nevertheless, ability in programming is becoming increasingly
essential to perform scientific analysis. Crystal Toolkit aims to tackle this
issue in two ways, by (1) allowing complex web apps to be constructed such
that the latest computational materials science algorithms can be used
directly via a web app, without any additional programming required by the
user, and (2) by providing a roadmap for scientists with some programming
training to make their own web apps to explore their own data. To achieve (2),
Crystal Toolkit should allow apps to be constructed and prototyped from just a
few lines of Python code. Python was chosen since it is now the de facto
standard programming language used for a wide range of scientific tasks, so a
language that a new scientist is most likely to have experience with.

  * •

Portability of representations between different media. Information can be
lost when converting from one representation of a given object to another. For
example, a scientist might use one form of plotting or visualization while
performing analysis, use another for a publication, and a third for
interactive use on a website (see Figure 4). Ideally, all of these views of
the same object should be the same, within the limitations of the respective
medium. This reduces the risk of loss of information and also reduces the
effort required to share data in different places; for example, by making it
easier to offer an interactive version of a visualization included in a
publication.

  * •

An open, inclusive attitude to new contributors. The Crystal Toolkit authors
want the framework to live and develop over time, and hope new developers will
join in this process. A Zenodo citation will also be provided to ensure new
developers can be acknowledged for their contributions, and all users of
Crystal Toolkit are encouraged to cite both this design manuscript and also
the Zenodo citation.

  * •

A tool for learning and developing intuition. In science, we are familiar with
how to think about the correspondence between reality and our models of
reality, where they agree and are helpful, and where the limits of our models
can obscure truth (for example, imagining a bond as a spring). However, with
the rise of computational science, we are less practiced in examining our
simulations, their predictions and our visual or textual representations of
them, whether in the form of graphics, animations or graphs. A good
visualization tool should help develop intuition for the models we use, and
act as an educational aid.

Figure 2: A design goal of Crystal Toolkit is to provide human-centered
representations of software objects, and specifically to ensure that the
default representations are as faithful to the underlying software object as
possible so as to help the user gain intuition about the use of these objects.
Figure 3: In a similar way that object-orientated programming allows complex
objects to be constructed and composed from simpler objects, visualizations in
Crystal Toolkit are designed to map directly to the underlying software object
and to be composable, so that complex visualizations can easily be built up.
This example shows part of how a visualization for silicon is constructed,
with the inset figure showing the final visualization after transformation to
a conventional cell and addition of an electron charge density layer. Figure
4: A significant motivation for the design of Crystal Toolkit is to allow
consistent views of the same information across journal publications, web
apps, and also during initial data generation and analysis, so as to reduce
the opportunity for loss of information or introduction of errors as the data
moves from one location to another. Publication figure is taken from Jiang et
al.[16]

##  III A Library of Ready-to-Use Components

Crystal Toolkit comes with several components included and ready to use. These
components map to existing data types used in the materials science community,
and mainly to specific classes available in the pymatgen[17] code. These
components can be used individually, or can be linked together; for example, a
“symmetry” component can be linked to a “crystal structure” component so that
symmetry information is always shown for the same crystal structure being
visualized.

The following is only an initial list of components, with components expected
to evolve as development proceeds and new components are made available, so
the current list should be considered a snapshot of Crystal Toolkit’s
capabilities at the time of publication. For an up-to-date list, please
consult the Crystal Toolkit documentation[18]. See Figure 5, left panel, for
illustrations of several of these components.

  1. 1.

Crystal Structures. Visualize crystal structures with a variety of
visualization options, including different colour schemes (including
accessible colour schemes), choice of atom size and bonding algorithms[19].
The structure visualizer also allows transformation to different
crystallographic settings and downloading of crystal structures to common file
formats including CIF. Maps to Structure and StructureGraph objects.

  2. 2.

Molecules. Visualize molecules in a similar way to crystal structures. Maps to
Molecule and MoleculeGraph objects.

  3. 3.

Band Structures. Visualize electronic or phonon band structures and densities
of states. Interact with the band structure by zooming in to specific regions
or, in future, by selecting different k-path conventions[20]. Maps to
BandStructure and DOS objects.

  4. 4.

Pourbaix Diagrams. Visualize Pourbaix (aqueous stability) diagrams built from
experimental and theoretical data. Interact by changing ion concentrations, or
show a heatmap of free energy for a specific material across voltage-pH space.
Maps to PourbaixDiagram objects.

  5. 5.

X-ray Absorption Spectra. Visualize X-ray absorption spectra, or other kinds
of spectra, and interact by applying Gaussian broadening. Maps to Spectrum
objects.

  6. 6.

Phase Diagrams. Visualize phase diagrams generated with a convex hull
construction. Supports 1-, 2-, 3- and 4-element phase diagrams, with 3-element
phase diagrams viewable in both 2D and 3D with $z$-axis of formation enthalpy,
and 4-element phase diagrams in 3D. Viewing in chemical potential space is
also possible. Interaction includes transformation into a grand potential
phase diagram by inclusion of open elements. Maps to PhaseDiagram objects.

  7. 7.

Symmetry. Show symmetry information, including space group, point group and
detected Wyckoff labels, for a given crystal structure, based on spglib[21].
Interaction includes being able to change tolerances for detection of
symmetry, an important consideration for computationally-obtained crystal
structures. Improvements include integration with Auguste[22] to show minimum
strain values.

  8. 8.

Diffraction Patterns. This component accepts a crystal structure and can
generate either an X-ray powder diffraction pattern or a transmission electron
microscope single crystal diffraction pattern. In both cases, the input
crystal structure is first transformed into its conventional setting such that
the peaks and diffraction spots respectively are annotated with the
conventional lattice indices. Interaction includes the ability to apply
Scherrer broadening to estimate finite size effects.

  9. 9.

Wulff Shapes. Visualize a Wulff shape and interact to examine specific facets
and surface energies. Maps to WulffShape objects. In an app, it is recommended
to link this component to a crystal structure component to allow corresponding
surfaces to be visualized on interaction with a given facet.

  10. 10.

Molecular Graphs. Molecular or crystal graphs, representing atoms as nodes and
bonds (or other information) as edges, can be represented as interactive
force-directed graphs. Maps to StructureGraph or MoleculeGraph objects.

  11. 11.

Periodic Table. This component provides a means to allow selection of specific
elements, to be used as an input for another component, and also supports
displaying a heatmap for a set of data associated with a list of elements.

  12. 12.

Brillouin Zones. Visualize Brillouin zones and k-paths in 3D. Improvements
include linking to band structure components to allow a given high-symmetry
line in a band structure diagram to be highlighted in the corresponding
Brillouin zone. Plotting of Fermi surfaces in the Brillouin zone is also
possible using IFermi[23].

  13. 13.

Transformations Component. This component accepts a crystal structure as input
and provides a means to transform that crystal structure using pymatgen. Maps
to Structure and AbstractTransformation objects.

##  IV Open-Source Development Model

Crystal Toolkit is developed as Open Source Software under a BSD license[24],
with the intent to encourage involvement of as many interested scientists as
possible in its development. Discussions on features and bugs are conducted
openly, currently on GitHub, and are open for participation from anyone.

This approach is motivated by the belief that scientific tools should be open
to allow their results to be reproducible, and to try to avoid issues of older
tools being abandoned as developers move on and then, over time, ceasing to
function. It is essential that new tools are developed with sustainability in
mind, so that they can continue to work into the future as the development
team or lead maintainers change.

The technical implementation of Crystal Toolkit is expected to change over
time as new technology becomes available. Thus, the current manuscript is
primarily focused on the design goals and overall philosophy of the project.

At time of writing, the project uses both Python code and JavaScript code, and
is built upon the Dash framework[25] by Plotly, with additional custom
components written using React[26] and packaged as Dash components[27]. React
is a JavaScript library for creating modern web apps that has now seen
extensive use by the web development community.

Dash was chosen specifically because it allowed an interactive web app to be
written in just a single file in the Python programming language yet provide
common web developer convenience features such as hot-module reloading on file
changes. This enables apps to be prototyped and developed more rapidly and
helps catch bugs more quickly compared to having to write code “in the dark”
without immediate feedback. All web components, including custom React
components written in JavaScript, map back to automatically-generated and
documented Python classes, such that they can be used by Python developers
without prior web development experience. However, the ability to write new,
custom React components empowers dedicated web developers to extend the
functionality of this code in useful ways (e.g., the development of the
Periodic Table component). Such a parallel development model enables web
experts to extend the code while materials science domain experts can apply
and use the code, without either blocking the other’s work. Furthermore, since
no client state is stored on the server by default, Dash readily scales
horizontally and is appropriate even for web apps which experience a large
amount of traffic like the Materials Project website.

These technologies were ultimately chosen to achieve the design goal of
improving accessibility of the project to newcomers: the Python programming
language has seen broad adoption in the scientific community and is now often
the first programming language that a new scientist will learn.

Examples on how to get started with Crystal Toolkit to write your own app,
including small, self-contained examples and installation instructions, are
available in the public documentation[18].

##  V Case Study: Powering the Materials Project

The Materials Project[28] started as a result of the Materials Genome
Initiative[29] with a mandate, among other goals, to develop “open web-based
access to computed information on materials and analysis tools to design novel
materials.” To date, it is now in use by over a quarter of a million
registered users from academia, industry and government, with a user base that
is growing rapidly year-by-year. The Materials Project offers an encyclopedic
view of most known, and many hypothetical, inorganic crystalline materials and
their predicted properties, with a goal of accelerating the materials
discovery process.

In 2022, Materials Project launched its first major iteration of its web
platform (see screenshot in Figure 5, right panel) based upon the Crystal
Toolkit framework. Adopting Crystal Toolkit has allowed more rapid development
of new features and apps in Materials Project by opening up development to the
broader materials science community including graduate students. For example,
a new Catalysis Explorer app, making available information computed
information on various adsorbates and crystallographic surfaces, and a new MOF
Explorer app[30], making available information on over 20,000 metal–organic
frameworks and coordination polymers, have both been made possible by the
Crystal Toolkit framework, powered by data supplied by the Materials Project’s
“MPContribs” API.

Crystal Toolkit has also enabled the development of internal tools to examine
the Materials Project database as it is being built to include new data, so as
to more easily identify issues and perform quality control.

To demo the Crystal Toolkit framework, an app for the Materials Project
website has also been developed. The Crystal Toolkit app allows users to
upload their own crystal structures and perform transformations using
_pymatgen_ without having to write their own code. These transformations might
include creating a surface slab, assigning oxidation states, or building grain
boundaries. The Crystal Toolkit framework began development as an effort to
improve this app to include new transformation options, and as the benefit of
the approach became evident, the technology grew to encompass the full
Materials Project website frontend.

##  VI Conclusion

Crystal Toolkit is a new tool intended for computational materials scientists
to help visualize and share their data. It is designed to complement and build
upon existing efforts, and offers a modest advance in a few key respects (see
“Design Goals”) by addressing the evolving needs of the materials science
community.

Crystal Toolkit provides ways to visualize many common data types used in
materials science, and provides interactive widgets that can be linked
together and combined in a web app, as well as providing visualizations in
analysis tools such as Jupyter notebooks. Crystal Toolkit is proven at scale,
and is now powering Materials Project website, its apps, and visualizations
included therein. As a demonstration, images of all 146,000 crystal structures
available in Materials Project at time of writing have been generated using
Crystal Toolkit, with the intent of releasing images under a Creative Commons
license for download on Wikimedia.

In the future, we will look to continue to expand the visualizations
available, and the means of interacting with them. Despite advances, in many
ways modern computer visualizations are not that dissimilar to Hoffman’s
original ball-and-stick models. It is important that as new data becomes
available that we also reassess the best ways of visualizing this data, and
thereby develop new intuitions and find new ways of unlocking the secrets
hidden within this data.

Figure 5: Left, a summary of just some of the components available for use in
a Crystal Toolkit web app. Many components rely on underlying functionality
available in the _pymatgen_ code, and some functionality was specifically
added to _pymatgen_ to facilitate features in Crystal Toolkit. Right, an
example of real-world usage of some of these components in a Materials Project
“materials detail page”, here showing the detail page for wurtzite GaN for
this example, located at https://materialsproject.org/materials/mp-804.

##  VII Author Contributions

Authors listed alphabetically with the exception of MKH, JXS, AJ, KAP. MKH
initiated and led the project, prepared the manuscript and all figures,
developed the code, and is lead maintainer of the code. JXS contributed
initial support for Jupyter notebooks, Asymptote rendering, surface plotting,
initial support for axes, and various bug fixes, and offered testing and
feedback throughout. JB developed the reaction calculator example app. OC
developed functionality related to periodic tiling. FC improved the JavaScript
code, contributed features to 3D rendering and wrote the Periodic Table React
component. AMG contributed a component for Fermi surfaces. RG improved rending
of molecules. PH deployed the current Materials Project website based on
Crystal Toolkit. HHL contributed functionality for rendering migration graph
objects. MM contributed to the phase diagram, X-ray diffraction and X-ray
absorption components. J Montoya contributed the Pourbaix component. GM
improved functionality related to rendering of crystals with magnetic moments.
J Munro contributed band structure component. CTO wrote several custom React
components. SZ worked on electron diffraction component, supervised by CO. GP
helped with X-ray diffraction component and with general bug fixes and
improvements. JR helped with general repository maintenance, continuous
integration and bug fixes. SW developed POV-Ray integration. BW helped with
developing an example app for display of contributed data via MPContribs and
improvements to documentation. DW helped with initial integration within the
legacy Materials Project website. RY helped with bug fixes, display of
calculation task documents, and absorption components. AJ provided helpful
feedback. KAP provided helpful feedback, supervision and funding support.

##  VIII Acknowledgements

Shyue Ping Ong is acknowledged for permission to adopt the “Crystal Toolkit”
name. Shyam Dwaraknath is acknowledged for helpful conversations throughout
and contributions to the design. Tyler Huntington is acknowledged for
assisting DW in the initial integration of Crystal Toolkit with the Materials
Project. Andrew Rosen is acknowledged for feedback and testing of Crystal
Toolkit during development of an interactive web app for MOF data and for
helpful comments on the manuscript. David Waroquiers is acknowledged for
helpful discussions during the development of chemical environment components.
Rachel Woods-Robinson is acknowledged for helpful comments on the manuscript.
This work was supported by the U.S. Department of Energy, Office of Science,
Office of Basic Energy Sciences, Materials Sciences and Engineering Division
under Contract No. DE-AC02-05-CH11231 (Materials Project program KC23MP).

## References

  * Parolo  _et al._ [2015] P. D. B. Parolo, R. K. Pan, R. Ghosh, B. A. Huberman, K. Kaski, and S. Fortunato, Attention decay in science, J. Informetr. 9, 734 (2015).
  * Meine [2022] C. Meine, Molecules and croquet balls, in  _Models_, edited by S. de Chadarevian and N. Hopwood (Stanford University Press, Redwood City, 2022) pp. 242–275.
  * Kobayashi  _et al._ [2021] R. Kobayashi, T. P. Goumans, N. O. Carstensen, T. M. Soini, N. Marzari, I. Timrov, S. Poncé, E. B. Linscott, C. J. Sewell, G. Pizzi, _et al._ , Virtual computational chemistry teaching laboratories—hands-on at a distance, Journal of Chemical Education 98, 3163 (2021).
  * Lehtola and Karttunen [2022] S. Lehtola and A. J. Karttunen, Free and open source software for computational chemistry education, Wiley Interdisciplinary Reviews: Computational Molecular Science 12, e1610 (2022).
  * Kokalj [1999] A. Kokalj, XCrySDen—a new program for displaying crystalline structures and electron densities, Journal of Molecular Graphics and Modelling 17, 176 (1999).
  * Stukowski [2010] A. Stukowski, Visualization and analysis of atomistic simulation data with OVITO–the Open Visualization Tool, Modelling and Simulation in Materials Science and Engineering 18, 015012 (2010).
  * Momma and Izumi [2011] K. Momma and F. Izumi, VESTA 3 for three-dimensional visualization of crystal, volumetric and morphology data, Journal of Applied Crystallography 44, 1272 (2011).
  * Hanwell  _et al._ [2012] M. D. Hanwell, D. E. Curtis, D. C. Lonie, T. Vandermeersch, E. Zurek, and G. R. Hutchison, Avogadro: an advanced semantic chemical editor, visualization, and analysis platform, Journal of Cheminformatics 4, 17 (2012).
  * Humphrey  _et al._ [1996] W. Humphrey, A. Dalke, and K. Schulten, VMD: Visual molecular dynamics, Journal of Molecular Graphics 14, 33 (1996).
  * Hanson [2010] R. M. Hanson, Jmol – a paradigm shift in crystallographic visualization, Journal of Applied Crystallography 43, 1250 (2010).
  * Rego and Koes [2015] N. Rego and D. Koes, 3dmol.js: molecular visualization with WebGL, Bioinformatics 31, 1322 (2015).
  * Rose and Hildebrand [2015] A. S. Rose and P. W. Hildebrand, NGL Viewer: a web application for molecular visualization, Nucleic Acids Research 43, W576 (2015).
  * Li [2003] J. Li, AtomEye: an efficient atomistic configuration viewer, Modelling and Simulation in Materials Science and Engineering 11, 173 (2003).
  * Card  _et al._ [1983] S. K. Card, T. P. Moran, and A. Newell, eds.,  _Psychology of human/computer interaction_ (Lawrence Erlbaum Associates, Mahwah, NJ, 1983).
  * Franconeri  _et al._ [2021] S. L. Franconeri, L. M. Padilla, P. Shah, J. M. Zacks, and J. Hullman, The science of visual data communication: What works, Psychological Science in the Public Interest 22, 110 (2021), pMID: 34907835.
  * Jiang  _et al._ [2021] C.-M. Jiang, L. I. Wagner, M. K. Horton, J. Eichhorn, T. Rieth, V. F. Kunzelmann, M. Kraut, Y. Li, K. A. Persson, and I. D. Sharp, Metastable ta2n3 with highly tunable electrical conductivity via oxygen incorporation, Materials horizons 8, 1744 (2021).
  * Ong  _et al._ [2013] S. P. Ong, W. D. Richards, A. Jain, G. Hautier, M. Kocher, S. Cholia, D. Gunter, V. L. Chevrier, K. A. Persson, and G. Ceder, Python materials genomics (pymatgen): A robust, open-source python library for materials analysis, Computational Materials Science 68, 314 (2013).
  * Crystal Toolkit Development Team [2022] Crystal Toolkit Development Team, Crystal Toolkit Documentation (2022).
  * Pan  _et al._ [2021] H. Pan, A. M. Ganose, M. Horton, M. Aykol, K. A. Persson, N. E. Zimmermann, and A. Jain, Benchmarking coordination number prediction algorithms on inorganic crystal structures, Inorganic chemistry 60, 1590 (2021).
  * Munro  _et al._ [2020] J. M. Munro, K. Latimer, M. K. Horton, S. Dwaraknath, and K. A. Persson, An improved symmetry-based approach to reciprocal space path selection in band structure calculations, npj Computational Materials 6, 112 (2020).
  * Togo and Tanaka [2018] A. Togo and I. Tanaka, Spglib: a software library for crystal symmetry search, arXiv preprint arXiv:1808.01590 (2018).
  * Larsen  _et al._ [2020] P. M. Larsen, E. L. Pang, P. A. Parrilo, and K. W. Jacobsen, Minimum-strain symmetrization of bravais lattices, Physical Review Research 2, 013077 (2020).
  * Ganose  _et al._ [2021] A. M. Ganose, A. Searle, A. Jain, and S. M. Griffin, Ifermi: A python library for fermi surface generation and analysis, Journal of Open Source Software 6, 3089 (2021).
  * bsd [2017] 3-Clause BSD License (2017).
  * Plotly [2017] Plotly, Dash (2017).
  * Materials Project [2020] Materials Project, mp-react-components (2020).
  * Materials Project [2019] Materials Project, dash-mp-components (2019).
  * Materials Project [2011] Materials Project, Materials Project Website (2011).
  * Materials Genome Initiative [2011] Materials Genome Initiative, Materials Genome Initiative Website (2011).
  * Rosen  _et al._ [2022] A. S. Rosen, V. Fung, P. Huck, C. T. O’Donnell, M. K. Horton, D. G. Truhlar, K. A. Persson, J. M. Notestein, and R. Q. Snurr, High-throughput predictions of metal–organic framework electronic properties: theoretical challenges, graph neural networks, and data exploration, npj Computational Materials 8, 1 (2022).