# Learning Multi-view Molecular Representations with Structured and
Unstructured Knowledge

Yizhen Luo  Institute of AI Industry Research (AIR), Tsinghua
UniversityBeijingChina<EMAIL_ADDRESS>,  Kai Yang  Institute
of AI Industry Research (AIR), Tsinghua UniversityBeijingChina
<EMAIL_ADDRESS>,  Massimo Hong  Institute of AI Industry Research
(AIR), Tsinghua UniversityBeijingChina<EMAIL_ADDRESS>,  Xing
Yi Liu  Institute of AI Industry Research (AIR), Tsinghua
UniversityBeijingChina<EMAIL_ADDRESS>,  Zikun Nie  Institute of AI
Industry Research (AIR), Tsinghua UniversityBeijingChina
<EMAIL_ADDRESS>,  Hao Zhou  Institute of AI Industry Research
(AIR), Tsinghua UniversityBeijingChina<EMAIL_ADDRESS>and
Zaiqing Nie  Institute of AI Industry Research (AIR), Tsinghua
UniversityPharmolix Inc.BeijingChina<EMAIL_ADDRESS>
(2024)

###### Abstract.

Capturing molecular knowledge with representation learning approaches holds
significant potential in vast scientific fields such as chemistry and life
science. An effective and generalizable molecular representation is expected
to capture the consensus and complementary molecular expertise from diverse
views and perspectives. However, existing works fall short in learning multi-
view molecular representations, due to challenges in explicitly incorporating
view information and handling molecular knowledge from heterogeneous sources.
To address these issues, we present MV-Mol, a molecular representation
learning model that harvests multi-view molecular expertise from chemical
structures, unstructured knowledge from biomedical texts, and structured
knowledge from knowledge graphs. We utilize text prompts to model view
information and design a fusion architecture to extract view-based molecular
representations. We develop a two-stage pre-training procedure, exploiting
heterogeneous data of varying quality and quantity. Through extensive
experiments, we show that MV-Mol provides improved representations that
substantially benefit molecular property prediction. Additionally, MV-Mol
exhibits state-of-the-art performance in multi-modal comprehension of
molecular structures and texts. Code and data are available at
https://github.com/PharMolix/OpenBioMed.

Multi-view Molecular Representation Learning, Knowledge Graphs, Text Mining

††journalyear: 2024††copyright: rightsretained††conference: Proceedings of the
30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining ; August
25–29, 2024; Barcelona, Spain.††booktitle: Proceedings of the 30th ACM SIGKDD
Conference on Knowledge Discovery and Data Mining (KDD ’24), August 25–29,
2024, Barcelona, Spain††isbn: 979-8-4007-0490-1/24/08††doi:
10.1145/3637528.3672043††ccs: Applied computing Bioinformatics††ccs: Computing
methodologies Knowledge representation and reasoning††ccs: Computing
methodologies Natural language processing

##  1\. Introduction

(a) The multi-view characteristics of molecular expertise.

(b) The heterogeneous sources of molecular expertise.

Figure 1. An overview of molecular expertise. Molecular expertise covers
diverse views that share consensus and complementary information. It resides
within heterogeneous sources that vary in quality and quantity.

Understanding the properties and functions of small molecules is a pivotal
issue in various scientific applications such as chemistry (Becker et al.,
2001), biology (Sliwoski et al., 2014), and material design (Guo et al.,
2021). Due to the substantial annotation costs for molecules, molecular
representation learning (MRL), which aims at capturing molecular knowledge
from abundant unlabeled data, has attracted significant research attention.
Through self-supervised learning (Liu et al., 2021b) on various types of
molecular structures, such as 1D SMILES strings (Chithrananda et al., 2020;
Irwin et al., 2022), 2D graphs (Hu et al., 2020; You et al., 2020; Wang et
al., 2022b) or 3D conformations (Zhu et al., 2022; Stärk et al., 2022; Zhou et
al., 2023), MRL models have achieved great success in advancing our
understanding of molecules.

However, to further extend the application scope of MRL models, one is faced
with a critical problem: molecular expertise is complicated and multifaceted,
spanning diverse disciplines and views. It comprises consensus information
shared by multiple views, as well as complementary information within each
specific view (Chao and Sun, 2016). As exemplified in Figure 1(a), the
characteristics of Aspirin are identified through various perspectives. From a
microscopic view, the molecule comprises a benzene ring, an acetoxy group, and
a carboxyl group, resulting in slight water solubility and weak acidity. From
a macroscopic view, it exhibits a monoclinic crystal structure, influencing
physical properties such as boiling point and stability. From a biological
view, it inhibits the production of prostaglandins by targeting
cyclooxygenase, thus exhibiting anti-inflammatory and pain-killing functions
(Ugurlucan et al., 2012). A versatile MRL model is expected to generate view-
based representations to address the distinctions between different
application contexts. Unfortunately, existing works primarily focus on
learning a universal representation to capture the consensus information
across various views, and fall short in grasping the complementary
characteristics of each view.

More recently, the emergence of MRL methods incorporating molecular knowledge
from heterogeneous sources has opened novel avenues for addressing the multi-
view problem. These models focus on jointly comprehending molecular
structures, structured knowledge from knowledge bases, and unstructured
knowledge from biomedical texts. The integration of heterogeneous inputs is
accomplished either through language modeling objectives within a unified
model (Zeng et al., 2022; Liu et al., 2023d; Pei et al., 2023), or by
contrastive learning objectives with independent encoders (Su et al., 2022;
Liu et al., 2023b; Seidl et al., 2023). Innovated by the success of these
models, we aim to capture multi-view molecular expertise from these structured
and unstructured knowledge sources.

Nevertheless, there exist two challenges in learning multi-view molecular
representations. First, as illustrated in Figure 1(a), the MRL model should
incorporate view information explicitly into its representations to ensure its
adaptability in broad applications. However, prior MRL models integrate view
information implicitly with ’wrapped texts’ (Liu et al., 2023d; Pei et al.,
2023) or fine-tuning on downstream tasks, which compromises their
understanding of the consensus and complementary relationships between
molecular knowledge from different views. Besides, as shown in Figure 1(b), it
is essential to address the heterogeneity of information sources, including
molecular structures, biomedical texts, and knowledge graphs, which vary in
quality and quantity. Previous works treat structured and unstructured
knowledge indiscriminately by transforming knowledge graphs into texts.
However, this may introduce biases across different views due to the
imbalanced distribution of pre-training data.

In this work, we propose MV-Mol, a comprehensive framework for Multi-View
Molecular representation learning with structured and unstructured knowledge,
to address the aforementioned problems. To explicitly incorporate view
information, we utilize text prompts to capture the complementary and
consensus characteristics of different views for molecules. We leverage
Q-Former (Li et al., 2023a), a multi-modal fusion architecture, to extract
view-based molecular representations by jointly comprehending molecular
structures and view prompts. Then, we propose a two-stage pre-training
strategy to address the heterogeneity of structured and unstructured
knowledge. The first stage aligns molecular structures with large-scale, noisy
texts, extracting consensus information across comprehensive views. The second
stage incorporates high-quality, structured knowledge from knowledge graphs.
Drawing inspiration from knowledge-enhanced pre-training (Wang et al., 2021,
2022a), we treat relations as specific types of views and describe them with
texts. With contrastive and generative objectives, MV-Mol is endowed with the
ability to grasp complementary information within different views.

We show the superior performance of MV-Mol through fine-tuning across various
downstream tasks. Benefiting from view-based molecular representations and
two-stage pre-training, MV-Mol achieves an average of 1.24% absolute gains
over the state-of-the-art method Uni-Mol (Zhou et al., 2023) on molecule
property prediction (Wu et al., 2018). Additionally, MV-Mol exhibits a deeper
understanding of connections between molecular structures and texts. On cross-
modal retrieval, MV-Mol improves the top-1 retrieval accuracy by 12.9% on
average over the best-performing baselines. On cross-modal generation (Edwards
et al., 2022), MV-Mol also yields more accurate predictions, as validated by
qualitative and quantitative studies.

Our contributions are summarized as follows: (1) To the best of our knowledge,
MV-Mol is the first work that addresses the multi-view problem in molecular
representation learning. (2) We propose to incorporate view information
explicitly by jointly comprehending molecular structures and view prompts
within a fusion network. (3) We develop a two-stage training pipeline to
harvest multi-view molecular expertise from structured and unstructured
knowledge sources that vary in quality and quantity.

Figure 2. Model architecture and pre-training pipeline of MV-Mol. (a) MV-Mol
is composed of a view-based molecule encoder and a multi-modal decoder. (b)
The molecule branch of the view-based molecule encoder. (c) The text branch of
the view-based molecule encoder. (d) We perform cross-modal contrastive and
cross-modal matching for modality alignment. (c) We model relation as a
textual prompt that constrains molecular knowledge from a specific view, and
design knowledge graph embedding and knowledge graph completion objectives for
multi-view knowledge incorporation. Both branches are activated when the head
entity refers to a molecule.

##  2\. Related work

###  2.1. Multi-view Molecular Representation Learning

Multi-view representation learning (Li et al., 2018; Zheng et al., 2023) aims
to pursue a unified and comprehensive feature representation based on multi-
view data. Most MRL models treat different forms of molecular structures as
distinct views and model complementary and consensus information with
different pre-training objectives. For example, DVMP (Zhu et al., 2023) treats
1D SMILES sequences and 2D molecular graphs as dual views. GraphMVP (Liu et
al., 2021a) treats 2D topologies and 3D geometries as two views, and proposes
independent objectives for intra-view and inter-view modeling. MORN (Ma et
al., 2022) treats the sequential, topological, and spatial information as
different views and implements a two-stage adaptive learning strategy with
multi-view fusion for molecular property prediction. (Wan et al., 2023) treats
atoms, scaffolds, and functional groups as independent views and proposes a
prompt-based aggregation module to extract task-specific molecular
representations. In this work, we extend the multi-view concept of molecules
as their inherent properties and patterns within a certain context, which is
defined flexibly by human-understandable textual prompts.

###  2.2. Molecular Representation Learning with Heterogeneous Sources

Recently, MRL approaches that jointly harvest heterogeneous sources including
molecular structures, biomedical texts, and knowledge graphs have surged.
Drawing parallels from vision-language pre-training (VLP) (Radford et al.,
2021; Li et al., 2023a; Sun et al., 2023), these models can be categorized as
follows: (1) Generative models including KV-PLM (Zeng et al., 2022), MolT5
(Edwards et al., 2022), MolXPT (Liu et al., 2023d), Text+Chem (Christofidellis
et al., 2023), GIMLET (Zhao et al., 2024a), BioMedGPT (Luo et al., 2023b),
Mol-Instructions (Fang et al., 2023a), BioT5 (Pei et al., 2023) and BioT5+
(Pei et al., 2024). These models treat molecules and texts as two languages
and jointly process them within a unified language model (Kenton and
Toutanova, 2019; Raffel et al., 2020; Radford et al., [n. d.]). (2)
Contrastive models like MoMu (Su et al., 2022), MoleculeSTM (Liu et al.,
2023b), CLAMP (Seidl et al., 2023), KANO (Fang et al., 2023b) and GODE (Jiang
et al., 2023). These models incorporate texts or knowledge graphs as cross-
modal supervision signals and augment molecular representations with
contrastive learning (Radford et al., 2021). (3) Hybrid models including MolFM
(Luo et al., 2023a), MolCA (Liu et al., 2023a), GIT-Mol (Liu et al., 2023c)
and 3D-MoLM (Li et al., 2024). These models perform fine-grained feature
integration with multi-modal fusion networks and incorporate both generative
and contrastive objectives for pre-training. While MV-Mol also adopts a fusion
network architecture, we distinguish it from prior works by our explicit
modeling of views by jointly comprehending molecular structures and texts and
our pre-training process that addresses data imbalance of heterogeneous
knowledge sources.

##  3\. Method

As depicted in Figure 2, MV-Mol is designed to learn multi-view molecular
representations from heterogeneous sources. The core of our approach lies in
modeling view information with text prompts.

The remainder of this section is organized as follows: First, we provide a
roadmap for extracting multi-view molecular expertise within heterogeneous
sources from a knowledge-enhanced pre-training perspective (Sec. 3.1). Then,
we introduce the model architecture for MV-Mol (Sec. 3.2) that extracts view-
based molecular representations. To address the heterogeneity of pre-training
data, we develop a two-stage pre-training paradigm, harvesting unstructured
knowledge from biomedical documents through modality alignment (Sec. 3.3) and
structured knowledge from knowledge bases with knowledge incorporation (Sec.
3.4).

###  3.1. A Knowledge-enhanced Pre-training Perspective for Multi-view MRL

In this section, we interpret multi-view MRL from the perspective of
knowledge-enhanced pre-training (Wang et al., 2021, 2022a; Hu et al., 2023),
providing a roadmap for methodology design. Inspired by OntoProtein (Zhang et
al., 2021), we formulate the pre-training data as triplets $(h,r,t)$, where
$h$ and $t$ are head and tail entities, and $r$ signifies the relation. We
define two types of entities, $E_{\text{mol}}$ and $E_{\text{text}}$.
$e^{(s)}\in E_{\text{mol}}$ is denoted as a molecule node with structural
information. $e^{(t)}\in E_{\text{text}}$ is denoted as a text node formulated
by a token sequence. $r$ is also formulated as text tokens that describe the
relation. Depending on the types of the head and tail entities, we further
categorize pre-training data as molecule-text triplets, molecule-molecule
triplets, and text-text triplets.

Knowledge-enhanced pre-training methods typically maximize the following
objective:

(1) $\mathcal{L}=S(f(h,r),g(t)),$

where $f(h,r)$ is the relation-transformed representation of the head entity,
$g(t)$ is the representation of the tail entity, and $S(\cdot,\cdot)$ is a
scoring function that measures the alignment of the triplet. We observe for
$h\in E_{\text{mol}}$, the view-based molecular representations have an
equivalent form of $f(h,r)$. Besides, structured knowledge graphs are
indispensable sources for multi-view MRL, as both views and relations can be
conceptualized as textual refinements over specific aspects of molecules.
Therefore, multi-view MRL comprises the following key design components:

  * •

The model architecture for $f(h,r)$, tasked with extracting different feature
representations of $h$ in the context of $r$. As $h$ and $r$ may come from
different modalities, the model should jointly comprehend molecular structures
and texts.

  * •

The formulation of the scoring function $S(\cdot,\cdot)$, intended to harvest
rich supervision signals from heterogeneous sources.

###  3.2. Model Architecture for MV-Mol

The model architecture of MV-Mol is depicted in Figure 2(a), comprising 2
parts: (1) A view-based molecule encoder that jointly comprehends molecular
structures and texts. (2) A multi-modal decoder that generates human-
understandable texts.

View-based Molecule Encoder. We define the structure of a molecule as
$\mathcal{M}=(\mathcal{V},\mathcal{E},\mathcal{C})$ where $\mathcal{V}$
represents atoms, $\mathcal{E}$ represents bonds, and
$\mathcal{C}\in\mathbb{R}^{|\mathcal{V}|\times 3}$ represents the 3D
coordinates for each atom. We leverage Uni-Mol (Zhou et al., 2023), a 15-layer
Transformer (Xu et al., 2018) pre-trained on 209M 3D conformations, to encode
$\mathcal{M}$. The pre-trained molecular model mitigates the
disproportionality between molecular structures and text data (Su et al.,
2022). The structure encoder $f_{\text{mol}}$ transforms $\mathcal{M}$ into
feature representations for each atom, which we denote as $z^{(a)}$.

In MV-Mol, we use text prompts to model different views of molecules,
represented by a sequence of $L$ tokens
$\mathcal{T}=[x_{1},x_{2},\cdots,x_{L}]$. Consequently, the view-based
molecule encoder is expected to comprehend molecular structures and texts
simultaneously. To achieve this, we leverage Querying Transformer (Q-Former)
(Li et al., 2023a), a novel multi-modal fusion architecture from vision-
language pre-training. The Q-Former consists of two branches, each consisting
of 12 bidirectional Transformer layers initialized with KV-PLM* (Zeng et al.,
2022). The molecule branch takes $K$ trainable query vectors as input
embeddings. The query features extract pivotal information from atomic
representations $z^{(a)}$ through cross-attention within every other
Transformer layer. The text branch digests $\mathcal{T}$ as its input. The
self-attention layers across the two branches are shared, allowing the queries
to grasp fine-grained semantics from text prompts. In this way, the molecular
representations are propagated within each Q-Former layer to incorporate
information from different views.

Overall, the view-based molecule encoder $f_{v}$ can transform the molecular
structure to a fixed number of features with the structure encoder and the
molecule branch of Q-Former, which is illustrated in Fig. 2b:

(2)
$z^{(s)}=\left[z_{1}^{(s)},z_{2}^{(s)},\cdots,z_{K}^{(s)}\right]=f_{v}(\mathcal{M}).$

Using Q-Former’s text branch, it can also comprehend natural language as in
Fig. 2c:

(3)
$z^{(t)}=\left[z_{1}^{(t)},z_{2}^{(t)},\cdots,z_{L}^{(t)}\right]=f_{v}(\mathcal{T}),$

Most importantly, as shown in Figure 2a, it can grasp view-based molecular
representations through multi-modal feature fusion of molecular structures and
texts:

(4)
$z^{(s,t)}=\left[z_{1}^{(s,t)},z_{2}^{(s,t)},\cdots,z_{K}^{(s,t)}\right]=f_{v}(\mathcal{M},\mathcal{T}).$

Multi-modal Decoder. The multi-modal decoder allows MV-Mol to interpret the
view-based molecular representations with natural language. We employ BioT5
(Pei et al., 2023), a molecular language model with 12 Transformer encoder
layers and 12 Transformer decoder layers. The model has undergone multi-task
pre-training on SELFIES strings of molecular structures, FASTA sequences of
proteins, and biomolecule-related texts. For MV-Mol, we leverage the decoder
branch of BioT5 as the text decoder $f_{\text{dec}}$, which transforms the
Q-Former outputs into a sequence of $N$ tokens $y=[y_{1},y_{2},\cdots,y_{N}]$
by casual generation:

(5)
$y_{i}=\text{argmax}\left\\{P\left(y_{i}\big{|}y_{<i},z^{(\cdot)}\right)\right\\},$

where $y_{<i}$ denotes preceding tokens, $z^{(\cdot)}$ refers to either
$z^{(s)}$, $z^{(t)}$ or $z^{(s,t)}$, and the conditional probability is
modeled by the text decoder $f_{\text{dec}}$.

Discussions. While our model shares a similar architecture with MolCA (Liu et
al., 2023a) and 3D-MoLM (Li et al., 2024), we emphasize that MV-Mol extracts
view-based molecular representations in Eq. 4 with both branches of the
Q-Former. In contrast, prior works are unaware of view information and encode
molecules as in Eq. 2, where the text branch of the Q-Former is deactivated.

###  3.3. Modality Alignment with Large-scale Unstructured Knowledge

In this stage, we aim to capture the consensus information within numerous
views of molecules and cultivate MV-Mol’s capability of jointly comprehending
molecular structures and texts. This is accomplished by aligning the feature
space between molecular structures $h\in E_{\text{mol}}$ and the relevant
contexts $t\in E_{\text{text}}$ from biomedical literature. We omit the
relation $r$ due to the unaffordable costs of annotating millions of noisy
texts. As shown in Figure 2b, the pre-training objectives are:

Cross-Modal Contrastive Loss. This objective maximizes the mutual information
between the structural and textual representations. We define structure-text
similarity as follows:

(6)
$\text{sim}\left(z^{(s)},z^{(t)}\right)=\max_{i=1,2,\cdots,K}\left\\{f_{\text{proj}}\left(z_{i}^{(s)}\right)^{T}f_{\text{proj}}\left(z^{(t)}_{1}\right)\right\\},$

where $f_{\text{proj}}(\cdot)$ is a projection network composed of a fully-
connected layer and $l_{2}$ normalization, and $z^{(t)}_{1}$ denotes the
textual representation of the [CLS] token.

The objective function is calculated as:

(7) $\displaystyle\mathcal{L}_{\text{cmc}}=-\frac{1}{2}$
$\displaystyle\left[\log\frac{\exp(\text{sim}[f_{v}(h),f_{v}(t)]/\tau)}{\sum_{h^{\prime}}\exp(\text{sim}[f_{v}(h^{\prime}),f_{v}(t)]/\tau)}\right.$
$\displaystyle+$
$\displaystyle\left.\log\frac{\exp(\text{sim}[f_{v}(h),f_{v}(t)]/\tau)}{\sum_{t^{\prime}}\exp(\text{sim}[f_{v}(h),f_{v}(t^{\prime})]/\tau)}\right],$

where $h^{\prime}$ and $t^{\prime}$ are molecules and texts within the same
mini-batch, and $\tau$ is a temperature hyper-parameter.

Cross-Modal Matching Loss. This objective fosters the fine-grained
comprehension of molecular structures and texts by predicting whether they
correspond to the same molecule. Following (Li et al., 2021, 2023a), we
acquire hard negatives for each sample by obtaining one misaligned structure
for the text and one misaligned text for the structure from the same mini-
batch based on the similarity scores defined in Eq. 6. The objective is
calculated as follows:

(8) $\mathcal{L}_{\text{cmm}}=H\left[y_{\text{cmm}}(h,t),\
f_{\text{cmm}}\left(f_{v}(h,t)\right)\right],$

where $f_{\text{cmm}}$ is a predictor composed of an average pooling and a
fully-connected layer, $y_{\text{cmm}}(\cdot,\cdot)$ is the ground truth label
indicating the alignment of the molecule-text pair, and $H(\cdot,\cdot)$
denotes cross-entropy.

The overall objective is the summation of the two loss functions:

(9)
$\mathcal{L}_{1}=\mathbb{E}_{(h,t)\sim\mathcal{D}_{1}}(\mathcal{L}_{\text{cmc}}+\mathcal{L}_{\text{cmc}}),$

where $\mathcal{D}_{1}$ is the pre-training dataset.

###  3.4. Multi-view Knowledge Incorporation with Heterogeneous Triplets

In this stage, we aim to incorporate high-quality structured knowledge into
MV-Mol and capture the multi-view characteristics of molecules within three
different types of triplets. Motivated by (Wang et al., 2021, 2022a), we
introduce relation transformed representations of the head entity,
$z^{(h,r)}$, calculated as follows:

(10) $z^{(h,r)}=\begin{cases}f_{v}(h,r),\quad\ \ \ h\in E_{mol}\\\
f_{v}(h\oplus r),\quad h\in E_{text}\end{cases}$

where $\oplus$ denotes concatenation. By modeling relations as view prompts,
we capture view-based molecular representations within the first term of the
equation. We connect $z^{(h,r)}$ and $t$ with objectives shown in Figure 2c,
which we detail as follows:

Knowledge Graph Embedding Loss. This objective captures the global features
within knowledge bases by aligning $z^{(h,r)}$ and the representations of the
tail entity. We first employ the following contrastive objective:

(11) $\displaystyle\mathcal{L}_{\text{kge\\_c}}=-\frac{1}{2}$
$\displaystyle\left[\log\frac{\exp(\text{sim}_{\text{tri}}[z^{(h,r)},f_{v}(t)]/\tau)}{\sum_{h^{\prime}}\exp(\text{sim}_{\text{tri}}[z^{(h^{\prime},r)},f_{v}(t)]/\tau)}\right.$
$\displaystyle+$
$\displaystyle\left.\log\frac{\exp(\text{sim}_{\text{tri}}[z^{(h,r)},f_{v}(t)]/\tau)}{\sum_{t^{\prime}}\exp(\text{sim}_{\text{tri}}[z^{(h,r)},f_{v}(t^{\prime})]/\tau)}\right],$

where $h^{\prime}$ and $t^{\prime}$ are head and tail entities within the same
batch, and $\text{sim}_{\text{tri}}(\cdot,\cdot)$ calculates the
representation similarity, which we detail in Appendix A. This objective is a
generalized form of Eq. 7 that incorporates relation information from
heterogeneous triplets.

Then, we incorporate a matching loss similar to Eq. 8 for molecule-text and
text-text triplets with the same negative sampling strategy for the head and
tail entity. We omit molecule-molecule triplets as $f_{v}$ cannot process two
molecules simultaneously. The objective is calculated as follows:

(12) $\mathcal{L}_{\text{kge\\_m}}=H\left[y_{\text{cmm}}(h,r,t),\
f_{\text{cmm}}\left(f_{v}(h,r\oplus t)\right)\right].$

Knowledge Graph Completion Loss. This objective elicits MV-Mol to ’talk’ about
molecular knowledge from a specific view. We implement the language model-
based tail entity prediction task (Qin et al., 2021; Liu et al., 2023e) by
feeding the relation-transformed representations $z^{(h,r)}$ into the text
decoder to generate descriptions for the tail entity. The objective is
calculated for molecule-text and text-text triplets in an auto-regressive
manner:

(13) $\mathcal{L}_{\text{kgc}}=\sum_{x_{i}\in
t}H\left[x_{i},P\left(x_{i}|x_{<i},z^{(h,r)}\right)\right],$

where the conditional probability is modeled by the text decoder
$f_{\text{dec}}$.

The overall objective is calculated as follows:

(14)
$\mathcal{L}_{2}=\mathbb{E}_{(h,r,t)\in\mathcal{D}_{2}}(\mathcal{L}_{\text{kge\\_c}}+\mathcal{L}_{\text{kge\\_m}}+\mathcal{L}_{\text{kgc}}),$

where $\mathcal{D}_{2}$ is the knowledge graph.

Table 1. AUROC scores for molecular property prediction on MoleculeNet. The best results are marked in bold, and the second-best results are underlined. $\uparrow$: the higher the better. ${\dagger}$: Our implementation. -: Not reported in the original paper. w/o view: without view prompts. Model | BBBP $\uparrow$ | Tox21 $\uparrow$ | ToxCast $\uparrow$ | SIDER $\uparrow$ | ClinTox $\uparrow$ | MUV $\uparrow$ | HIV $\uparrow$ | BACE $\uparrow$ | Avg $\uparrow$
---|---|---|---|---|---|---|---|---|---
GraphCL (You et al., 2020) | 67.5±3.3 | 75.0±0.3 | 62.8±0.2 | 60.1±1.3 | 78.9±4.2 | 77.1±1.0 | 75.0±0.4 | 68.7±7.8 | 70.64
GraphMVP (Liu et al., 2021a) | 72.4±1.6 | 74.4±0.2 | 63.1±0.4 | 63.9±1.2 | 77.5±4.2 | 75.0±1.0 | 77.0±1.2 | 81.2±0.9 | 73.07
GEM (Fang et al., 2022) | 72.4±0.4 | 78.1±0.1 | 69.2±0.4 |  67.2±0.4 | 90.1±1.3 |  81.7±0.5 | 80.6±0.9 | 85.6±1.1 | 78.11
Uni-Mol (Zhou et al., 2023) | 72.9±0.6 |  79.6±0.5 |  69.6±0.1 | 65.9±1.3 | 91.9±1.8 | 82.1±1.3 |  80.8±0.3 | 85.7±0.2 | 78.56
KV-PLM (Zeng et al., 2022) | 66.9±1.1 | 64.7±1.8 | 58.6±0.4 | 55.3±0.8 | 84.3±1.5 | 60.2±2.9 | 68.8±4.9 | 71.9±2.1 | 66.29
MoMu (Su et al., 2022) | 70.5±2.0 | 75.6±0.3 | 63.4±0.5 | 60.5±0.9 | 79.9±4.1 | 70.5±1.4 | 75.9±0.8 | 76.7±2.1 | 71.63
MoleculeSTM (Liu et al., 2023b) | 70.0±0.5 | 77.0±0.4 | 65.1±0.3 | 61.0±1.0 |  92.5±1.1 | 73.4±2.9 | 76.9±1.8 | 80.8±1.3 | 74.57
MolCA (Liu et al., 2023a) | 70.0±0.5 | 77.2±0.5 | 64.5±0.8 | 63.0±1.7 | 89.5±0.7 | - | - | 79.8±0.5 | -
GIT-Mol (Liu et al., 2023c) | 73.9±0.6 | 75.9±0.5 | 66.8±0.5 | 63.4±0.8 | 88.3±1.2 | - | - | 81.1±1.5 | -
BioT5† (Pei et al., 2023) | 75.2±0.1 | 75.5±0.3 | 64.4±0.1 | 62.4±0.1 | 85.8±0.6 | 75.9±0.7 | 79.7±0.4 | 89.2±0.1 | 76.01
MV-Mol (w/o view) | 73.0±0.2 | 79.7±0.1 | 69.7±0.2 | 65.3±0.2 | 93.2±0.2 | 81.6±0.4 | 79.9±0.5 | 87.3±0.2 | 78.71
MV-Mol | 73.6±0.2 | 80.3±0.6 | 70.0±0.4 | 67.3±0.0 | 95.6±1.6 | 82.1±0.5 | 81.4±0.3 | 88.2±0.4 | 79.80
Table 2. Statistics of benchmark datasets. We present the number of molecules, data samples, prediction tasks, and split protocols. Dataset | # Molecules | # Samples | # Tasks | Split
---|---|---|---|---
BBBP | 2,039 | 2,039 | 1 | Scaffold (8/1/1)
Tox21 | 7,831 | 7,831 | 12
ToxCast | 8,597 | 8,597 | 617
SIDER | 1,427 | 1,427 | 27
ClinTox | 1,478 | 1,478 | 2
MUV | 93.807 | 93,807 | 17
HIV | 41,127 | 41,127 | 1
BACE | 1,513 | 1,513 | 1
PCdes | 11,112 | 11,112 | 1 | Scaffold (7/1/2)
MVST | 7,102 | 16,996 | 1
CheBI-20 | 33,008 | 33,008 | 1 | Random (8/1/1)

##  4\. Experiments

In this section, we conduct extensive experiments to demonstrate the
effectiveness of the proposed model by answering the following questions:

  * •

Q1. Compared to prior MRL models, does MV-Mol provide better molecular
representations?

  * •

Q2. Does MV-Mol address the heterogeneity of molecular structures and texts?

  * •

Q3. Do view-based molecular representations capture the consensus and
complementary knowledge from different views? How do view prompts affect the
representations?

  * •

Q4. How do the design components within the two pre-training stages affect MV-
Mol?

###  4.1. Pre-training Setup

Pre-training Dataset. For the modality alignment stage, we collect large-scale
molecule-text pairs by named entity recognition (NER) (Nadeau and Sekine,
2007) and entity linking (Kolitsas et al., 2018) on 3.5M scientific
publications following previous works (Zeng et al., 2022; Liu et al., 2023d;
Pei et al., 2023), and obtain a total of 60K molecules and 12M molecule-text
pairs. For the knowledge incorporation stage, we construct a knowledge graph
by combining public databases including CheBI (Degtyarenko et al., 2007),
PubChem (Kim et al., 2016), and DrugBank (Wishart et al., 2018), which
comprises 273K entities and 643K relations. During pre-training data
collection, we exclude molecules within the test set of downstream datasets to
alleviate information leakage. More details are presented in Appendix B.

Training Configurations. MV-Mol comprises a molecular structure encoder with
47.3M parameters, a Q-Former with 180.8M parameters, and a text decoder with
140.2M parameters. We pre-train MV-Mol for 70K steps with a batch size of 256
for the modality alignment stage and 50K steps with a batch size of 192 for
the knowledge incorporation stage. The pre-training procedure is performed on
4 NVIDIA A100 GPUs for 5 days. We use the AdamW (Loshchilov and Hutter, 2018)
optimizer with a weight decay of $5\times 10^{-2}$. The learning rate is
linearly warmed up to $10^{-4}$ in the first 2K steps and then decreases to
$10^{-5}$ following a cosine annealing strategy. The temperature $\tau$ for
contrastive learning is fixed as $0.1$.

###  4.2. Molecular Property Prediction (Q1)

To assess the quality of molecular representations, we perform fine-tuning
experiments on molecular property prediction, a widely adopted task in MRL.

Datasets. We adopt 8 classification datasets from MoleculeNet (Wu et al.,
2018), a popular benchmark that covers diverse properties of molecules. In
line with prior works (Liu et al., 2021a; Zhou et al., 2023), we adopt
Scaffold split with a train/validation/test ratio of 8/1/1. Statistics of the
benchmark datasets are presented in Table 2.

Baselines. We adopt two types of baseline MRL models:

  * •

Uni-modal MRL baselines: GraphCL (You et al., 2020), GraphMVP (Liu et al.,
2021a), GEM (Fang et al., 2022) and UniMol (Zhou et al., 2023). These models
are pre-trained solely with molecular structures including 2D topologies, 3D
conformations, or both.

  * •

Multi-modal MRL baselines: KV-PLM (Zeng et al., 2022), MoMu (Su et al., 2022),
Mole-culeSTM (Liu et al., 2023b), MolCA (Liu et al., 2023a), GIT-Mol (Liu et
al., 2023c), and BioT5 (Pei et al., 2023). These models incorporate
heterogeneous pre-training data including molecular structures and texts.

Implementation Details. We manually write descriptions for each dataset and
feed the prompt and molecular structure into MVMol’s encoder to obtain view-
based molecular representations. Then, we feed the encoder outputs into a max-
pooling layer and a projector for binary classification. The projector is
composed of two fully connected layers with ReLU activation. We also implement
a variant of our model named MV-Mol (w/o view) by removing view prompts and
solely encoding molecules as in Eq. 2. Notably, we apply the prediction head
with BioT5’s encoder for a fair comparison of the representation quality. We
perform experiments three times with different random seeds and report AUROC
scores. Refer to Appendix D for more details of our hyper-parameters and
prompts.

Results and Analysis. The overall results of molecular property prediction are
displayed in Table 1. The key observations are as follows: (1) Compared with
the state-of-the-art model Uni-Mol, MV-Mol achieves an absolute gain of 1.24%
on average. Overall, MV-Mol performs best on 5 of 8 datasets and second-best
on the remaining 3 datasets. This outstanding performance validates the
informativeness of MV-Mol’s molecular representations. (2) On SIDER, ClinTox,
and BACE which consist of limited training samples, MV-Mol attains more
significant improvements over Uni-Mol (1.4% on SIDER, 3.7% on ClinTox, and
2.5% on BACE), which indicates that adapting molecular representations with
view prompts brings more significant benefits to low-data scenarios. (3)
Removing view descriptions leads to a decline of 0.79% on average. This
performance drop corroborates our claims that downstream applications require
molecular knowledge from different views, and a universal molecular
representation fails to capture the specific context of each task.

###  4.3. Zero-shot Cross-modal Retrieval (Q2)

Table 3. Zero cross-modal retrieval results on the test split of PCdes. The best results are marked in bold, and the second-best results are underlined. $\uparrow$: the higher the better. SubTask | Model |  MRR $\uparrow$ |  R@1 $\uparrow$ |  R@5 $\uparrow$ |  R@10 $\uparrow$
---|---|---|---|---|---
S-T |  CLAMP (Seidl et al., 2023) | 0.58 | 0.18 | 0.49 | 0.67
MoMu (Su et al., 2022) | 11.10 | 5.93 | 15.20 | 20.46
MoleculeSTM (Liu et al., 2023b) | 40.30 | 26.99 | 55.33 | 66.48
MolCA (Liu et al., 2023a) | 33.65 | 23.21 | 45.52 | 54.29
3D-MoLM (Li et al., 2024) | 1.68 | 0.53 | 1.61 | 2.57
MV-Mol (w/o view) | 41.17 | 30.85 | 53.12 | 60.23
MV-Mol | 45.25 | 36.66 | 55.78 | 60.41
T-S |  CLAMP (Seidl et al., 2023) | 0.43 | 0.08 | 0.27 | 0.54
MoMu (Su et al., 2022) | 11.58 | 6.29 | 15.78 | 21.86
MoleculeSTM (Liu et al., 2023b) | 30.18 | 17.45 | 43.94 | 57.71
MolCA (Liu et al., 2023a) | 33.48 | 23.70 | 44.30 | 53.30
3D-MoLM (Li et al., 2024) | 2.69 | 0.76 | 3.60 | 5.89
MV-Mol (w/o view) | 45.51 | 35.89 | 56.68 | 62.70
MV-Mol | 46.54 | 37.24 | 57.17 | 62.97
Table 4. Zero cross-modal retrieval results on the test split of MVST. The best results are marked in bold, and the second-best results are underlined. $\uparrow$: the higher the better. SubTask | Model |  MRR $\uparrow$ |  R@1 $\uparrow$ |  R@5 $\uparrow$ |  R@10 $\uparrow$
---|---|---|---|---|---
S-T |  CLAMP (Seidl et al., 2023) | 0.34 | 0.07 | 0.23 | 0.30
MoMu (Su et al., 2022) | 13.59 | 6.97 | 20.06 | 26.69
MoleculeSTM (Liu et al., 2023b) | 17.80 | 9.61 | 25.98 | 35.59
MolCA (Liu et al., 2023a) | 18.89 | 14.25 | 24.43 | 27.33
3D-MoLM (Li et al., 2024) | 1.09 | 0.19 | 1.32 | 2.22
MV-Mol (w/o view) | 24.87 | 16.40 | 34.08 | 41.32
MV-Mol | 35.38 | 24.28 | 47.92 | 57.27
T-S |  CLAMP (Seidl et al., 2023) | 0.33 | 0.04 | 0.23 | 0.53
MoMu (Su et al., 2022) | 11.81 | 5.16 | 17.60 | 25.26
MoleculeSTM (Liu et al., 2023b) | 12.68 | 4.86 | 19.41 | 29.67
MolCA (Liu et al., 2023a) | 11.05 | 4.86 | 18.13 | 22.43
3D-MoLM (Li et al., 2024) | 1.44 | 0.26 | 2.00 | 3.09
MV-Mol (w/o view) | 20.86 | 9.35 | 33.37 | 44.26
MV-Mol | 34.07 | 22.85 | 45.73 | 56.56

We investigate whether MV-Mol captures rich molecular knowledge from
heterogeneous sources including texts and knowledge graphs. Hence, we evaluate
MV-Mol on cross-modal retrieval, which contains two sub-tasks: structure-to-
text (S-T) retrieval and text-to-structure (T-S) retrieval. The former aims to
retrieve the most relevant text that describes a given molecule, and the
latter aims to retrieve a molecule that best fits the textual description.

Datasets. We incorporate two datasets: PCdes (Zeng et al., 2022) and MVST. The
PCdes dataset is collected from the CheBI database and comprises 15K molecules
with their biochemical definitions and property descriptions. MVST (Multi-View
Structure-Text) is a novel dataset introduced by this work. It is collected
from chemical, physical, and pharmacokinetic definitions in PubChem. It
consists of 7.1K molecules, each corresponding with 2 or more texts from
different views. Both datasets are partitioned by Scaffold split with a
train/validation/test ratio of 7/1/2. Differing from prior works (Zeng et al.,
2022; Su et al., 2022), we perform retrieval on the whole test set and report
MRR (mean reversed rank) and Recall at 1/5/10. More details of the datasets
are presented in Appendix C.

Baselines. We compare MV-Mol with models that have undergone contrastive
learning on molecule-text data including CLAMP (Seidl et al., 2023), MoMu (Su
et al., 2022), MoleculeSTM (Liu et al., 2023b), MolCA (Liu et al., 2023a) and
3D-MoLM (Li et al., 2024).

Implementation Details. On PCdes, we fix the view description as ’biochemical
properties and functions’. On MVST, we write descriptions for physical,
chemical, and pharmacokinetic views to generate different representations for
the same molecule. To further improve the retrieval performance, we modify the
re-ranking algorithm in (Li et al., 2021) with an ensemble technique. We first
retrieve the top-$k$ candidates based on structure-text similarity in Eq. 6.
Then, we calculate the CMM logits in Eq. 8 for these $k$ candidates. Finally,
we re-rank them by a linear combination of the structure-text similarity
scores and the CMM logits.

Results and Analysis. The retrieval results on PCDes are presented in Tab. 3.
We observe that: (1) MV-Mol outperforms state-of-the-art methods by 4.95% and
13.06% in MRR for S-T and T-S retrieval. While MoleculeSTM achieves the best
results in R@10 on S-T retrieval, it is worth noting that the model is pre-
trained on texts similar to those in PCDes and molecules within the test set,
raising information leakage concerns. Nevertheless, MV-Mol demonstrates
effectiveness in bridging molecular structures and texts. (2) Adding the view
description yields a notable gain of 3.08% and 1.03% in MRR for S-T and T-S
retrieval, indicating that the view-based molecular representations align
better with texts in PCdes.

Table 4 shows the results on MVST, where MV-Mol yields substantial
improvements, surpassing MoleculeSTM and MolCA by 16.49% and 21.39% in MRR for
S-T and T-S retrieval. The greater performance gain suggests that MV-Mol is
endowed with molecular knowledge from a wider range of views. Notably,
removing view descriptions leads to a more significant decrease, which we
attribute to the multi-view property of the MVST dataset. While a molecule may
correspond with multiple descriptions from distinct views, MV-Mol could
distinguish between them based on the view-based molecular representations.

###  4.4. Cross-modal Generation (Q2)

Table 5. Molecule captioning results on the test split of ChEBI-20. ’BL’ is short for ’BLEU’, and ’R’ is short for ’ROUGE’. The best results are marked in bold, and the second-best results are underlined. $\uparrow$: the higher the better. -: Not reported in the original paper. Model | #Params |  BL-2 $\uparrow$ |  BL-4 $\uparrow$ |  R-1 $\uparrow$ |  R-2 $\uparrow$ |  R-L $\uparrow$ |  METEOR $\uparrow$
---|---|---|---|---|---|---|---
MolReGPT (Li et al., 2023b) | - | 0.565 | 0.482 | 0.623 | 0.450 | 0.543 | 0.585
MolT5-base (Edwards et al., 2022) | 250M | 0.540 | 0.457 | 0.634 | 0.485 | 0.578 | 0.569
MolT5-large (Edwards et al., 2022) | 770M | 0.594 | 0.508 | 0.654 | 0.510 | 0.594 | 0.612
MoMu (Su et al., 2022) | 770M | 0.599 | 0.515 | - | - | 0.593 | 0.597
MolXPT (Liu et al., 2023d) | 350M | 0.594 | 0.505 | 0.660 | 0.511 | 0.597 | 0.626
Text+Chem (Christofidellis et al., 2023) | 250M | 0.625 | 0.542 | 0.682 | 0.543 | 0.622 | 0.648
MolCA (Liu et al., 2023a) | 110M | 0.620 | 0.531 | 0.681 | 0.537 | 0.618 | 0.651
ChatMol (Zeng et al., 2023) | 220M | 0.620 | 0.535 | 0.677 | 0.538 | 0.617 | 0.644
GIT-Mol (Liu et al., 2023c) | 320M | 0.352 | 0.263 | 0.575 | 0.485 | 0.560 | 0.533
BioT5 (Pei et al., 2023) | 252M | 0.635 | 0.556 | 0.692 | 0.559 | 0.633 | 0.656
MV-Mol (w/o view) | 452M | 0.650 | 0.572 | 0.698 | 0.567 | 0.640 | 0.669
MV-Mol | 452M | 0.653 | 0.575 | 0.699 | 0.569 | 0.640 | 0.669

Cross-modal generation also involves two sub-tasks: structure-to-text
generation (molecule captioning) and text-to-structure generation (text-based
molecule generation).

Datasets. We conduct experiments on ChEBI-20 (Edwards et al., 2021), a widely-
adopted dataset for both tasks. We follow the original split with a
train/valid/test ratio of 8/1/1 and adopt evaluation metrics in (Edwards et
al., 2022).

Baselines. We compare MV-Mol with the following baselines:

  * •

MolReGPT (Li et al., 2023b). This baseline leverages a kNN few-shot strategy
(Nie et al., 2022) for in-context learning (Dong et al., 2022) with ChatGPT
(OpenAI, 2022).

  * •

Molecular language model baselines, including MolT5 (Edwards et al., 2022),
MolXPT (Liu et al., 2023d), Text+Chem (Christofidellis et al., 2023), ChatMol
(Zeng et al., 2023) and BioT5 (Pei et al., 2023) that process 1D SMILES or
SELFIES strings of molecules and texts within a unified language model.

  * •

Fusion model baselines, namely MoMu (Su et al., 2022). The baseline connects
the GNN outputs with MolT5 to solve molecule captioning.

Implementation Details. For molecule captioning, we fix the view description
as ’biochemical properties and functions’. In line with (Su et al., 2022; Liu
et al., 2023c), we concatenate the outputs of the MV-Mol encoder with the
SELFIES representations of the BioT5 encoder and feed the results into our
multi-modal decoder to generate the caption. For text-based molecule
generation, we feed the textual representations in Eq. 3 into the MV-Mol
decoder to generate the SELFIES string of the molecule.

Results and Analysis. Table 5 shows the results of molecule captioning.
Compared to the state-of-the-art model BioT5, MV-Mol shows 1.8% improvements
in BLEU scores (Papineni et al., 2002) and 1.3% improvements in METEOR
(Banerjee and Lavie, 2005) scores, indicating that it generates smoother and
more semantically related molecular descriptions.

The performance comparison for text-based molecule generation is reported in
Table 5. MV-Mol achieves state-of-the-art performance in most evaluation
metrics, highlighted by an outstanding exact ratio of 0.438, surpassing state-
of-the-art by 2.5%. While MV-Mol underperforms Text+Chem in fingerprint
similarity, we argue that these metrics are only calculated on valid
molecules, which may yield over-optimistic results.

Above all, the experiments show that MV-Mol addresses data heterogeneity and
can flexibly translate between molecular structures and natural language.

###  4.5. A Deeper Analysis of View-based Molecular Representations (Q3)

Visualization of View-based Molecular Representations. To investigate how view
descriptions affect molecular representations, we visualize the molecular
features of MV-Mol on the MVST dataset with UMAP (McInnes et al., 2018). As
shown in Figure 3, the molecular representations based on chemical, physical,
and pharmacokinetic views exhibit clear separation along the x-axis.
Furthermore, we highlight three representative molecules (indicated by colored
shapes) along with corresponding texts from each view (indicated by black
shapes), which are closely positioned. Notably, the partial order of the three
molecules along the y-axis remains consistent under different views. These
observations support our intuition that view-based molecular representations
can concurrently capture the consensus and complementary molecular knowledge
from different views.

Figure 3. Visualization of view-based molecular representations. We show the
molecular representations from chemical, physical, and pharmacokinetic views.
We also highlight three molecules and the representations of their textual
descriptions from each view.

View Prompt Engineering. Motivated by recent advances in prompt engineering
(Chen et al., 2023; White et al., 2023), we explore how different view prompts
affect the representations of our model. We perform analysis on BBBP, SIDER,
ClinTox, and BACE from molecule property prediction, and implement the
following strategies:

  * •

Empty prompt. This strategy refers to MV-Mol (w/o view), where the view
information is not provided.

  * •

Word prompt. We use the name of the dataset as the prompt.

  * •

Sentence prompt. We manually write a brief definition of the prediction task
as the prompt.

  * •

Paragraph prompt. We write a well-rounded introduction to the dataset.

As shown in Figure 4(b), the sentence prompt works best with BBBP, SIDER, and
BACE, while the paragraph prompt yields the best results on ClinTox. We posit
that BBBP, SIDER, and BACE focus on specific properties including blood-brain
barrier penetration, adverse drug reactions, and target inhibition, where a
single sentence suffices to encapsulate all relevant information. In ClinTox,
however, more texts are necessary to describe clinical trial criteria and
toxicity measures. Besides, the word prompt brings little improvement over the
empty prompt. We attribute this to the abbreviated dataset names, which are
not frequently used in the pre-training corpora and provide little meaningful
information for MV-Mol.

(a) Table 5: Text-based molecule generation results on the test split of ChEBI-20. The best results are marked in bold, and the second-best results are underlined. $\uparrow$: The higher the better. $\downarrow$: The lower the better. -: Not reported in the original paper. Model | # Params |  BLEU $\uparrow$ |  Exact $\uparrow$ |  Valid $\uparrow$ |  Levenshtein $\downarrow$ |  MACCS FTS $\uparrow$ |  RDKit FTS $\uparrow$ |  Morgan FTS $\uparrow$ |
---|---|---|---|---|---|---|---|---|---
MolReGPT (Li et al., 2023b) | - | 0.790 | 0.139 | 0.887 | 24.910 | 0.847 | 0.708 | 0.624 |
MolT5-base (Edwards et al., 2022) | 250M | 0.779 | 0.082 | 0.786 | 25.188 | 0.787 | 0.661 | 0.601 |
MolT5-large (Edwards et al., 2022) | 770M | 0.854 | 0.311 | 0.905 | 16.071 | 0.834 | 0.746 | 0.684 |
MoMu (Su et al., 2022) | 235M | 0.815 | 0.183 | 0.863 | 20.520 | 0.847 | 0.737 | 0.678 |
MolXPT (Liu et al., 2023d) | 350M | - | 0.215 | 0.983 | - | 0.859 | 0.757 | 0.667 |
Text+Chem (Christofidellis et al., 2023) | 250M | 0.853 | 0.322 | 0.943 | 16.870 | 0.901 | 0.816 | 0.757 |
ChatMol (Zeng et al., 2023) | 220M | 0.848 | 0.258 | 0.947 | 16.759 | 0.883 | 0.790 | 0.726 |
GIT-Mol (Liu et al., 2023c) | 250M | 0.756 | 0.051 | 0.928 | 26.315 | 0.738 | 0.582 | 0.519 |
BioT5 (Pei et al., 2023) | 252M | 0.854 | 0.413 | 1.000 | 15.200 | 0.886 | 0.801 | 0.734 |
MV-Mol | 252M | 0.858 | 0.438 | 1.000 | 14.952 | 0.890 | 0.810 | 0.745 |

(b) Figure 4: Experiment results on BBBP, SIDER, ClinTox, and BACE for
different prompts.

###  4.6. Ablation Studies (Q4)

To explore the contribution of our two-stage pre-training, we compare the
performances of MV-Mol by removing different pre-training objectives for uni-
modal and multi-modal tasks and display the results in Table 6. We observe
that: (1) Knowledge graph completion (KGC) loss mainly contributes to cross-
modal generation but compromises MV-Mol’s performance on cross-modal retrieval
(line 2). (2) Removing the knowledge incorporation stage leads to significant
deterioration on all downstream tasks (line 3), highlighting the significance
of incorporating multi-view knowledge of molecules. (3) The cross-modal
matching objective also benefits all tasks (lines 3 and 4), as it cultivates
the joint comprehension of molecules and texts. (4) Removing the modality
alignment stage leads to moderate performance drops (line 5), indicating the
indispensability of molecular knowledge from large-scale publications. (5)
Compared with Stage 1 pre-training, Stage 2 brings more contributions,
especially for retrieval tasks, which we attribute to two reasons. First, the
knowledge incorporation stage is the key to multi-view MRL, allowing MV-Mol to
jointly comprehend view prompts and molecular structures. Second, the
knowledge graph enjoys higher quality than the noisy molecule-text pairs for
the modality alignment stage.

Table 6. Comparison between MV-Mol with different pre-training objectives. For molecule property prediction, we report the average results on 8 datasets. For zero-shot cross-modal retrieval, we report the average results of Recall at 1/5/10 on the MVST dataset. For cross-modal generation, we report BLEU-2. Stage 1 | Stage 2 | Property | Retrieval | Generation
---|---|---|---|---
CMC | CMM | KGE | KGC | S-T | T-S | S-T
✔ | ✔ | ✔ | ✔ | 0.798 | 0.432 | 0.417 | 0.653
✔ | ✔ | ✔ | ✗ | 0.796 | 0.437 | 0.451 | 0.646
✔ | ✔ | ✗ | ✗ | 0.786 | 0.306 | 0.260 | 0.647
✔ | ✗ | ✗ | ✗ | 0.781 | 0.275 | 0.248 | 0.644
✗ | ✗ | ✔ | ✔ | 0.787 | 0.395 | 0.400 | 0.650

##  5\. Limitations and broader impacts

While our work presents promising results in molecular representation
learning, several areas remain for future exploration: (1) Improving the scale
and quality of the pre-training data for molecules from structured and
unstructured knowledge sources. (2) Exploring the scaling laws of foundation
models (Kaplan et al., 2020) by incorporating large language models (LLMs)
(Touvron et al., 2023; OpenAI, 2023) into MV-Mol. (3) Applying MV-Mol to more
biomedical entities such as proteins (Xu et al., 2023), DNA and RNA sequences
(Richard et al., 2024), and cell transcriptomics (Zhao et al., 2024b).

MV-Mol bears promise for accelerating biomedical research. However, there is a
concern that MV-Mol may be misused to generate potentially dangerous or toxic
molecules. Therefore, it is essential to ensure the responsible and ethical
use of the model. We emphasize that MV-Mol should be employed solely for
research purposes, and any potential medical applications of MV-Mol should
undergo comprehensive experimental evaluations.

##  6\. Conclusion

In this paper, we present MV-Mol, a molecular representation learning model to
harness multi-view molecular expertise from structured and unstructured
knowledge sources. To capture the consensus and complementary information
across different views, we propose to model views explicitly with textual
prompts. We leverage a multi-modal fusion architecture to extract view-based
molecular representations, and address the heterogeneity of structured and
unstructured knowledge with varying quality and quantity with a two-stage pre-
training strategy. Through extensive experiments, we demonstrate the
superiority of MV-Mol on molecular property prediction and cross-modal
translation. Under thorough analysis aimed at safety, MV-Mol has the potential
to deliver unprecedented advancements to the biomedical research community.

###### Acknowledgements.

This research is supported by the National Key R&D Program of China (No.
2022YFF1203002) and PharMolix Inc.

## References

  * (1)
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_. 65–72.
  * Becker et al. (2001) Oren M Becker, Alexander D MacKerell Jr, Benoit Roux, and Masakatsu Watanabe. 2001.  _Computational biochemistry and biophysics_.  Crc Press.
  * Chao and Sun (2016) Guoqing Chao and Shiliang Sun. 2016.  Consensus and complementarity based maximum entropy discrimination for multi-view classification.  _Information Sciences_ 367 (2016), 296–310.
  * Chen et al. (2023) Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, and Shengxin Zhu. 2023.  Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review.  _arXiv preprint arXiv:2310.14735_ (2023).
  * Chithrananda et al. (2020) Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. 2020.  ChemBERTa: large-scale self-supervised pretraining for molecular property prediction.  _arXiv preprint arXiv:2010.09885_ (2020).
  * Christofidellis et al. (2023) Dimitrios Christofidellis, Giorgio Giannone, Jannis Born, Ole Winther, Teodoro Laino, and Matteo Manica. 2023.  Unifying molecular and textual representations via multi-task language modelling.  _arXiv preprint arXiv:2301.12586_ (2023).
  * Degtyarenko et al. (2007) Kirill Degtyarenko, Paula De Matos, Marcus Ennis, Janna Hastings, Martin Zbinden, Alan McNaught, Rafael Alcántara, Michael Darsow, Mickaël Guedj, and Michael Ashburner. 2007.  ChEBI: a database and ontology for chemical entities of biological interest.  _Nucleic acids research_ 36, suppl_1 (2007), D344–D350.
  * Delmas et al. (2021) Maxime Delmas, Olivier Filangi, Nils Paulhe, Florence Vinson, Christophe Duperier, William Garrier, Paul-Emeric Saunier, Yoann Pitarch, Fabien Jourdan, Franck Giacomoni, et al. 2021\.  building a Knowledge Graph from public databases and scientific literature to extract associations between chemicals and diseases.  _Bioinformatics_ 37, 21 (2021), 3896–3904.
  * Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022.  A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_ (2022).
  * Edwards et al. (2022) Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. 2022.  Translation between Molecules and Natural Language. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_. 375–413.
  * Edwards et al. (2021) Carl Edwards, ChengXiang Zhai, and Heng Ji. 2021.  Text2mol: Cross-modal molecule retrieval with natural language queries. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. 595–607.
  * Fang et al. (2022) Xiaomin Fang, Lihang Liu, Jieqiong Lei, Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. 2022.  Geometry-enhanced molecular representation learning for property prediction.  _Nature Machine Intelligence_ 4, 2 (2022), 127–134.
  * Fang et al. (2023a) Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. 2023a.  Mol-Instructions-A Large-Scale Biomolecular Instruction Dataset for Large Language Models. In _The Twelfth International Conference on Learning Representations_.
  * Fang et al. (2023b) Yin Fang, Qiang Zhang, Ningyu Zhang, Zhuo Chen, Xiang Zhuang, Xin Shao, Xiaohui Fan, and Huajun Chen. 2023b.  Knowledge graph-enhanced molecular contrastive learning with functional prompt.  _Nature Machine Intelligence_ 5, 5 (2023), 542–553.
  * Guo et al. (2021) Kai Guo, Zhenze Yang, Chi-Hua Yu, and Markus J Buehler. 2021.  Artificial intelligence and machine learning in design of mechanical materials.  _Materials Horizons_ 8, 4 (2021), 1153–1172.
  * Hu et al. (2023) Linmei Hu, Zeyi Liu, Ziwang Zhao, Lei Hou, Liqiang Nie, and Juanzi Li. 2023.  A survey of knowledge enhanced pre-trained language models.  _IEEE Transactions on Knowledge and Data Engineering_ (2023).
  * Hu et al. (2020) W Hu, B Liu, J Gomes, M Zitnik, P Liang, V Pande, and J Leskovec. 2020.  Strategies For Pre-training Graph Neural Networks. In _International Conference on Learning Representations (ICLR)_.
  * Irwin et al. (2022) Ross Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum. 2022.  Chemformer: a pre-trained transformer for computational chemistry.  _Machine Learning: Science and Technology_ 3, 1 (2022), 015022.
  * Jiang et al. (2023) Pengcheng Jiang, Cao Xiao, Tianfan Fu, and Jimeng Sun. 2023.  Bi-level Contrastive Learning for Knowledge Enhanced Molecule Representations.  (2023).
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ (2020).
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of NAACL-HLT_. 4171–4186.
  * Kim et al. (2016) Sunghwan Kim, Paul A Thiessen, Evan E Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He, Siqian He, Benjamin A Shoemaker, et al. 2016\.  PubChem substance and compound databases.  _Nucleic acids research_ 44, D1 (2016), D1202–D1213.
  * Kolitsas et al. (2018) Nikolaos Kolitsas, Octavian-Eugen Ganea, and Thomas Hofmann. 2018.  End-to-End Neural Entity Linking. In _Proceedings of the 22nd Conference on Computational Natural Language Learning_. 519–529.
  * Landrum et al. (2013) Greg Landrum et al. 2013\.  RDKit: A software suite for cheminformatics, computational chemistry, and predictive modeling.  _Greg Landrum_ 8 (2013), 31.
  * Li et al. (2023a) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a.  BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.  (2023).
  * Li et al. (2023b) Jiatong Li, Yunqing Liu, Wenqi Fan, Xiao-Yong Wei, Hui Liu, Jiliang Tang, and Qing Li. 2023b.  Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective.  _arXiv preprint arXiv:2306.06615_ (2023).
  * Li et al. (2021) Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021.  Align before fuse: Vision and language representation learning with momentum distillation.  _Advances in neural information processing systems_ 34 (2021), 9694–9705.
  * Li et al. (2024) Sihang Li, Zhiyuan Liu, Yanchen Luo, Xiang Wang, Xiangnan He, Kenji Kawaguchi, Tat-Seng Chua, and Qi Tian. 2024.  Towards 3D Molecule-Text Interpretation in Language Models.  _arXiv preprint arXiv:2401.13923_ (2024).
  * Li et al. (2018) Yingming Li, Ming Yang, and Zhongfei Zhang. 2018.  A survey of multi-view representation learning.  _IEEE transactions on knowledge and data engineering_ 31, 10 (2018), 1863–1883.
  * Liu et al. (2023c) Pengfei Liu, Yiming Ren, and Zhixiang Ren. 2023c.  Git-mol: A multi-modal large language model for molecular science with graph, image, and text.  _arXiv preprint arXiv:2308.06911_ (2023).
  * Liu et al. (2023b) Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang, Chaowei Xiao, and Animashree Anandkumar. 2023b.  Multi-modal molecule structure–text model for text-based retrieval and editing.  _Nature Machine Intelligence_ 5, 12 (2023), 1447–1457.
  * Liu et al. (2021a) Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. 2021a.  Pre-training Molecular Graph Representation with 3D Geometry. In _International Conference on Learning Representations_.
  * Liu et al. (2021b) Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. 2021b.  Self-supervised learning: Generative or contrastive.  _IEEE transactions on knowledge and data engineering_ 35, 1 (2021), 857–876.
  * Liu et al. (2023e) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2023e.  GPT understands, too.  _AI Open_ (2023).
  * Liu et al. (2023a) Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. 2023a.  MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter. In _The 2023 Conference on Empirical Methods in Natural Language Processing_.
  * Liu et al. (2023d) Zequn Liu, Wei Zhang, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Ming Zhang, and Tie-Yan Liu. 2023d.  MolXPT: Wrapping Molecules with Text for Generative Pre-training.  _arXiv preprint arXiv:2305.10688_ (2023).
  * Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S Weld. 2020.  S2ORC: The Semantic Scholar Open Research Corpus. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 4969–4983.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Decoupled Weight Decay Regularization. In _International Conference on Learning Representations_.
  * Luo et al. (2023a) Yizhen Luo, Kai Yang, Massimo Hong, Xingyi Liu, and Zaiqing Nie. 2023a.  Molfm: A multimodal molecular foundation model.  _arXiv preprint arXiv:2307.09484_ (2023).
  * Luo et al. (2023b) Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. 2023b.  Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine.  _arXiv preprint arXiv:2308.09442_ (2023).
  * Ma et al. (2022) Runze Ma, Yidan Zhang, Xinye Wang, Zhenyang Yu, and Lei Duan. 2022.  MORN: Molecular Property Prediction Based on Textual-Topological-Spatial Multi-View Learning. In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_. 1461–1470.
  * McInnes et al. (2018) Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. 2018.  UMAP: Uniform Manifold Approximation and Projection.  _Journal of Open Source Software_ 3, 29 (2018), 861.
  * Nadeau and Sekine (2007) David Nadeau and Satoshi Sekine. 2007.  A survey of named entity recognition and classification.  _Lingvisticae Investigationes_ 30, 1 (2007), 3–26.
  * Nie et al. (2022) Feng Nie, Meixi Chen, Zhirui Zhang, and Xu Cheng. 2022.  Improving few-shot performance of language models via nearest neighbor calibration.  _arXiv preprint arXiv:2212.02216_ (2022).
  * OpenAI (2022) OpenAI. 2022.  Introducing chatgpt.  https://doi.org/blog/chatgpt
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_. 311–318.
  * Pei et al. (2024) Qizhi Pei, Lijun Wu, Kaiyuan Gao, Xiaozhuan Liang, Yin Fang, Jinhua Zhu, Shufang Xie, Tao Qin, and Rui Yan. 2024.  BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning.  _arXiv preprint arXiv:2402.17810_ (2024).
  * Pei et al. (2023) Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingce Xia, and Rui Yan. 2023.  BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_. 1102–1123.
  * Qin et al. (2021) Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou. 2021.  ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 3350–3363.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International conference on machine learning_. PMLR, 8748–8763.
  * Radford et al. ([n. d.]) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. [n. d.].  Improving language understanding by generative pre-training.  ([n. d.]).
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ 21, 1 (2020), 5485–5551.
  * Richard et al. (2024) Guillaume Richard, Bernardo P de Almeida, Hugo Dalla-Torre, Christopher Blum, Lorenz Hexemer, Priyanka Pandey, Stefan Laurent, Marie P Lopez, Alexander Laterre, Maren Lang, et al. 2024\.  ChatNT: A Multimodal Conversational Agent for DNA, RNA and Protein Tasks.  _bioRxiv_ (2024), 2024–04.
  * Seidl et al. (2023) Philipp Seidl, Andreu Vall, Sepp Hochreiter, and Günter Klambauer. 2023.  Enhancing activity prediction models in drug discovery with the ability to understand human language.  _arXiv preprint arXiv:2303.03363_ (2023).
  * Sliwoski et al. (2014) Gregory Sliwoski, Sandeepkumar Kothiwale, Jens Meiler, and Edward W Lowe. 2014.  Computational methods in drug discovery.  _Pharmacological reviews_ 66, 1 (2014), 334–395.
  * Stärk et al. (2022) Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Liò. 2022.  3d infomax improves gnns for molecular property prediction. In _International Conference on Machine Learning_. PMLR, 20479–20502.
  * Su et al. (2022) Bing Su, Dazhao Du, Zhao Yang, Yujie Zhou, Jiangmeng Li, Anyi Rao, Hao Sun, Zhiwu Lu, and Ji-Rong Wen. 2022.  A molecular multimodal foundation model associating molecule graphs with natural language.  _arXiv preprint arXiv:2209.05481_ (2022).
  * Sun et al. (2023) Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. 2023.  Generative pretraining in multimodality.  _arXiv preprint arXiv:2307.05222_ (2023).
  * Swainston et al. (2016) Neil Swainston, Janna Hastings, Adriano Dekker, Venkatesh Muthukrishnan, John May, Christoph Steinbeck, and Pedro Mendes. 2016.  libChEBI: an API for accessing the ChEBI database.  _Journal of Cheminformatics_ 8 (2016), 1–6.
  * Tosco et al. (2014) Paolo Tosco, Nikolaus Stiefl, and Gregory Landrum. 2014.  Bringing the MMFF force field to the RDKit: implementation and validation.  _Journal of cheminformatics_ 6 (2014), 1–4.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023\.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ (2023).
  * Ugurlucan et al. (2012) Murat Ugurlucan, Ilker M Caglar, Fatma N Turhan Caglar, Sedat Ziyade, Oguzhan Karatepe, Yahya Yildiz, Ertugrul Zencirci, Funda Gungor Ugurlucan, Ahmet H Arslan, Semra Korkmaz, et al. 2012\.  Aspirin: from a historical perspective.  _Recent Patents on Cardiovascular Drug Discovery (Discontinued)_ 7, 1 (2012), 71–76.
  * Wan et al. (2023) Yue Wan, Jialu Wu, Tingjun Hou, Chang-Yu Hsieh, and Xiaowei Jia. 2023.  From molecules to scaffolds to functional groups: building context-dependent molecular representation via multi-channel learning.  _arXiv preprint arXiv:2311.02798_ (2023).
  * Wang et al. (2021) Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021.  KEPLER: A unified model for knowledge embedding and pre-trained language representation.  _Transactions of the Association for Computational Linguistics_ 9 (2021), 176–194.
  * Wang et al. (2022a) Xintao Wang, Qianyu He, Jiaqing Liang, and Yanghua Xiao. 2022a.  Language models as knowledge embeddings.  _arXiv preprint arXiv:2206.12617_ (2022).
  * Wang et al. (2022b) Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. 2022b.  Molecular contrastive learning of representations via graph neural networks.  _Nature Machine Intelligence_ 4, 3 (2022), 279–287.
  * White et al. (2023) Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023.  A prompt pattern catalog to enhance prompt engineering with chatgpt.  _arXiv preprint arXiv:2302.11382_ (2023).
  * Wishart et al. (2018) David S Wishart, Yannick D Feunang, An C Guo, Elvis J Lo, Ana Marcu, Jason R Grant, Tanvir Sajed, Daniel Johnson, Carin Li, Zinat Sayeeda, et al. 2018\.  DrugBank 5.0: a major update to the DrugBank database for 2018.  _Nucleic acids research_ 46, D1 (2018), D1074–D1082.
  * Wu et al. (2018) Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. 2018.  MoleculeNet: a benchmark for molecular machine learning.  _Chemical science_ 9, 2 (2018), 513–530.
  * Xu et al. (2018) Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018.  How Powerful are Graph Neural Networks?. In _International Conference on Learning Representations_.
  * Xu et al. (2023) Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. 2023.  Protst: Multi-modality learning of protein sequences and biomedical texts. In _International Conference on Machine Learning_. PMLR, 38749–38767.
  * You et al. (2020) Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020.  Graph contrastive learning with augmentations.  _Advances in neural information processing systems_ 33 (2020), 5812–5823.
  * Zeng et al. (2022) Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2022.  A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals.  _Nature communications_ 13, 1 (2022), 862.
  * Zeng et al. (2023) Zheni Zeng, Bangchen Yin, Shipeng Wang, Jiarui Liu, Cheng Yang, Haishen Yao, Xingzhi Sun, Maosong Sun, Guotong Xie, and Zhiyuan Liu. 2023.  Interactive Molecular Discovery with Natural Language.  _arXiv preprint arXiv:2306.11976_ (2023).
  * Zhang et al. (2021) Ningyu Zhang, Zhen Bi, Xiaozhuan Liang, Siyuan Cheng, Haosen Hong, Shumin Deng, Qiang Zhang, Jiazhang Lian, and Huajun Chen. 2021.  OntoProtein: Protein Pretraining With Gene Ontology Embedding. In _International Conference on Learning Representations_.
  * Zhao et al. (2024a) Haiteng Zhao, Shengchao Liu, Ma Chang, Hannan Xu, Jie Fu, Zhihong Deng, Lingpeng Kong, and Qi Liu. 2024a.  Gimlet: A unified graph-text model for instruction-based molecule zero-shot learning.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Zhao et al. (2024b) Suyuan Zhao, Jiahuan Zhang, Yizhen Luo, Yushuai Wu, and Zaiqing Nie. 2024b.  LangCell: Language-Cell Pre-training for Cell Identity Understanding.  _arXiv preprint arXiv:2405.06708_ (2024).
  * Zheng et al. (2023) Qinghai Zheng, Jihua Zhu, Zhongyu Li, Zhiqiang Tian, and Chen Li. 2023.  Comprehensive multi-view representation learning.  _Information Fusion_ 89 (2023), 198–209.
  * Zhou et al. (2023) Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. 2023.  Uni-Mol: a universal 3D molecular representation learning framework.  (2023).
  * Zhu et al. (2022) Jinhua Zhu, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. 2022.  Unified 2d and 3d pre-training of molecular representations. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_. 2626–2636.
  * Zhu et al. (2023) Jinhua Zhu, Yingce Xia, Lijun Wu, Shufang Xie, Wengang Zhou, Tao Qin, Houqiang Li, and Tie-Yan Liu. 2023.  Dual-view molecular pre-training. In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_. 3615–3627.

##  Appendix A Details of MV-Mol

The representation similarity $\text{sim}_{\text{tri}}(\cdot,\cdot)$ in Eq. 11
is calculated as follows:

(A1) $\displaystyle\text{sim}_{\text{tri}}(z^{(h,r)},z^{(t)})$
$\displaystyle=\begin{cases}\max_{i=1,2,\cdots,K}\left\\{f_{\text{proj}}(z^{(h,r)}_{i})^{T}f_{\text{proj}}(z^{(t)}_{1})\right\\},h\in
E_{\text{mol}},t\in E_{\text{text}},\\\
\max_{i,j=1,2,\cdots,K}\left\\{f_{\text{proj}}(z_{i}^{(h,r)})^{T}f_{\text{proj}}(z^{(t)}_{j})\right\\},h,t\in
E_{\text{mol}},\\\
f_{\text{proj}}(z^{(h,r)}_{1})^{T}f_{\text{proj}}(z^{(t)}_{1}),h,t\in
E_{\text{text}}\\\ \end{cases}$

##  Appendix B Pre-training Data Collection

Following (Zeng et al., 2022; Su et al., 2022), we perform named entity
recognition (NER) (Nadeau and Sekine, 2007) and entity linking (EL) (Kolitsas
et al., 2018) to obtain molecule-text pairs from biomedical literature.
Specifically, we first sample 321K frequently accessed molecules from PubChem
(Kim et al., 2016). Then we identify mentions of these molecules within 3.5M
publications from S2ORC (Lo et al., 2020). We start with the corresponding
sentence of the mention and randomly extend the context by the previous or
next sentence within the paragraph until the text snippet exceeds 256 tokens.
Consequently, we collect 12M molecule-text pairs for 60K molecules.

We also collect a knowledge graph for molecules by combining public databases
including PubChem (Kim et al., 2016), CheBI (Degtyarenko et al., 2007),
DrugBank (Wishart et al., 2018) and FORUM (Delmas et al., 2021). We first
extract 196K molecules from these databases and perform deduplication. Then,
we collect the connections between these molecules and other entities as
follows:

  * •

We collect drug property descriptions that span diverse views from PubChem. We
treat each description as a unique node and connect it with the corresponding
molecule.

  * •

We collect drug targets, drug enzymes, drug carriers, and drug transporters
from DrugBank. We also collect BindingDB compounds and protein targets with
$Ki\leq 10nM$.

  * •

We collect drug-disease and drug-ontology connections from FORUM with
$\text{q}\\_\text{value}<10^{-6}$.

  * •

We collect drug-ontology and ontology-ontology connections from CheBI with
libCheBI toolkit (Swainston et al., 2016).

In total, the knowledge graph comprises 273K entities and 643K relationships.
Table A1 presents the overall statistics of the knowledge graph. Following
(Zhou et al., 2023), we calculate 3D conformations by the MMFF algorithm
(Tosco et al., 2014) within the RDKit (Landrum et al., 2013) package for
molecules.

Table A1. Statistics of the collected knowledge graph. The left column denotes
the entity or relation type, and the right column denotes the number of
entities and relations. Entities
---
Molecules | 196,454
Targets | 403
Ontologies | 22,076
Diseases | 2,835
Properties | 51,882
All | 273,650
Relations
Molecule-Target Interaction | 23,870
Molecule-Ontology Connection | 252,555
Molecule-Disease Association | 172,241
Molecule-Physical Property | 6,166
Molecule-Chemical Property | 38,206
Molecule-Pharmacokinetic Property | 7,510
Molecule-Molecule Connection | 119,793
Ontology-Ontology Connection | 22,706
All | 643,047

##  Appendix C Benchmark Datasets

###  C.1. Molecular Property Prediction

We adopt 8 classification datasets for molecular property prediction,
including BBBP (Blood-Brain Barrier Penetration), Tox21 (Toxicology in the
21st Century), ToxCast (Toxicity), SIDER (Side Effects Resource), ClinTox
(Clinical Trial Outcomes and Toxicity), MUV (Maximum Unbiased Validation), HIV
(HIV inhibition), and BACE (Human $\beta$-secretase 1 Binding).

###  C.2. Cross-Modal Retrieval

For cross-modal retrieval, we incorporate the following datasets: PCDes. The
original dataset (Zeng et al., 2022) comprises 15K molecules and their
descriptions in CheBI. We filter out 3,888 molecules that have appeared in our
pre-training dataset. Unlike the original experiment setting, we adopt the
Scaffold split with a train/validation/test ratio of 7/1/2 and perform zero-
shot retrieval on the test set with the entire paragraph.

MVST. This dataset is introduced as part of our work. Different from PCDes,
each molecule within MVST corresponds to multiple descriptive texts,
distinguished by a given view. The dataset primarily focuses on 3 types of
views:

  * •

The chemical view. The data for this view is directly derived from the CheBI
description of molecules within the PubChem database.

  * •

The physical view. We assemble multiple columns within PubChem including the
general physical description, color, form, order, boiling point, melting
point, flashing point, solubility, density, vapor pressure, stability (shelf
life), and decomposition. The text is organized as ’COLUMN_NAME:
COLUMN_DESCRIPTION’. We randomly sample rows if the assembled text exceeds 256
tokens.

  * •

The pharmacokinetic view. We also incorporate multiple columns from PubChem
including drug indication, disposal methods, drug-food interactions, toxicity
summary, metabolism, therapeutic uses, inhalation systems, absorption,
distribution, exertion, biological half-life, adverse effects, and
carcinogenicity. The organization of the text and length control follows the
same as the physical view.

Following previous works (Edwards et al., 2022; Zeng et al., 2022), we replace
the name of the molecule within texts with ’the molecule’. After collecting
data from each view, we sample molecules with textual descriptions from 2 or
more views and partition them by Scaffold split with a train/validation/test
ratio of 7/1/2.

###  C.3. Cross-Modal Generation

We incorporate the CheBI-20 dataset, a widely adopted dataset collected from
the CheBI database. We follow the original random split with a
train/validation/test ratio of 8/1/1.

##  Appendix D Downstream Experiment Details

We manually write descriptions for each dataset in MoleculeNet as view prompts
to improve molecular representations. Specifically:

  * •

BBBP. The prompt is ’blood-brain barrier penetration (permeability)’.

  * •

Tox21. The prompt is ’Qualitative toxicity measurements including nuclear
receptors and stress response pathways’.

  * •

ToxCast. The prompt is ’Qualitative toxicity measurements’.

  * •

SIDER. The prompt is ’adverse drug reactions (ADR) for 27 system organ
classes’.

  * •

ClinTox. The prompt is ’Qualitative data of drugs if they failed clinical
trials for toxicity reasons’.

  * •

MUV. The prompt is ’Subset of PubChem BioAssay designed for validation of
virtual screening techniques’.

  * •

HIV. The prompt is ’Experimentally measured abilities to inhibit HIV
replication’.

  * •

BACE. The prompt is ’Binding results for human $\beta$-secretase 1 (BACE-1)’.