# Scientific Language Modeling: A Quantitative Review of Large Language Models
in Molecular Science

Pengfei Liu
Peng Cheng Laboratory
School of Computer Science and Engineering,
Sun Yat-sen University
Jun Tao
School of Computer Science and Engineering,
Sun Yat-sen University
Zhixiang Ren
Peng Cheng Laboratory
<EMAIL_ADDRESS>
Corresponding author

###### Abstract

Efficient molecular modeling and design are crucial for the discovery and
exploration of novel molecules, and the incorporation of deep learning methods
has revolutionized this field. In particular, large language models (LLMs)
offer a fresh approach to tackle scientific problems from a natural language
processing (NLP) perspective, introducing a research paradigm called
scientific language modeling (SLM). However, two key issues remain: how to
quantify the match between model and data modalities and how to identify the
knowledge-learning preferences of models. To address these challenges, we
propose a multi-modal benchmark, named ChEBI-20-MM, and perform 1263
experiments to assess the model’s compatibility with data modalities and
knowledge acquisition. Through the modal transition probability matrix, we
provide insights into the most suitable modalities for tasks. Furthermore, we
introduce a statistically interpretable approach to discover context-specific
knowledge mapping by localized feature filtering. Our pioneering analysis
offers an exploration of the learning mechanism and paves the way for
advancing SLM in molecular science.

_K_ eywords Large Language Model $\cdot$ Molecular Science $\cdot$
Benchmarking

##  1 Introduction

Molecular modeling and design are pivotal in the discovery and development of
new molecules, serving a wide range of applications from drug discovery to
materials science. Nevertheless, traditional approaches to discovering new
molecules or refining existing ones are often arduous, costly, and fraught
with a high risk of failure. In contrast, modern computational approaches
significantly improve the creation and modification of molecules. These
techniques speed up the identification and optimization of drug candidates,
enhancing discovery efficiency. Yet, they require considerable computational
resources. Artificial Intelligence (AI) methods have made significant strides
in this domain. They offer faster computations and the ability to handle large
datasets. Various machine learning [1] and graph models [2] [3] [4] are
employed for molecular property prediction, while generative models [5] [6]
[7] are utilized for molecular design. However, most AI models often face
challenges in generalizability and flexibility, requiring extensive data for
training to ensure accuracy and require specific adjustments for each
molecular task.

Figure 1: The paradigm of the review. a. Molecular modeling and design tasks,
showcasing six task types with their standard modeling methods and data
examples. b. The processes of tasks, we divide common molecular data into two
categories: internal and external information. Internal information, integral
to molecular representation, can be converted through various tools. External
information is more accessible to human understanding. Additionally, this part
highlights the research scope of our review, detailing the input and output
for each task.

Transformers [8] offer an advantage with their robust text encoding and
generation capabilities. These models can be fine-tuned with minimal task-
specific adjustments, making them more versatile and efficient in molecular
modeling and design. Moreover, since the advent of ChatGPT [9] and GPT-4 [10],
LLMs have emerged as a groundbreaking trend, especially in molecular science.
LLMs, with their advanced capabilities in processing and generating human-like
text, present a novel paradigm for understanding and designing molecular
structures. Their ability to assimilate and analyze vast amounts of textual
data can provide unprecedented insights, overcoming some of the limitations of
traditional AI methods. This new capability combines accuracy and novelty for
improved outcomes, termed chemical knowledge. Its effectiveness depends on
factors like input data, model architecture, and training strategies. However,
current reviews and benchmark assessments of this capability are not
comprehensive.

Existing surveys in molecular science, like the molecule-generation survey
[11], often lack comprehensive model comparisons and have a limited task
scope. The knowledge-driven survey [12] categorizes molecular learning but
misses detailed method comparisons and dataset discussions. And recent
benchmarks, such as one testing ChatGPT [13], cover eight chemical tasks, each
providing unique chemical insights. Mol-Instructions [14] offers a dataset for
fine-tuning with various molecule and protein instructions, boosting
biomolecular understanding in LLMs. However, these surveys and benchmarks lack
multi-modal content and do not sufficiently explore the models’ chemical
knowledge.

In summary, this study comprehensively reviews Transformers and LLMs in
molecular modeling and design. We categorize six common molecular tasks into
three distinct objectives: description, embedding, and generation, as vividly
depicted in Figure 1. Moreover, we establish a unified multi-modal benchmark
ChEBI-20-MM and conduct experiments to assess the compatibility of data
modalities, model architectures, and diverse task types, examining their
impact on task performance. Additionally, our end-to-end visual method
showcases the discovery of modeling insights embedded with chemical knowledge.
Overall, our main contributions are the following:

  * •

This work analyzes LLMs in molecule modeling, categorizes existing models, and
presents a multi-modal benchmark (ChEBI-20-MM) for performance evaluation,
backed by 1263 experiments.

  * •

We analyze the modal transition probability matrix and identify best match
between different data modalities and model architectures.

  * •

We introduce a statistically interpretable approach to showcase the knowledge
acquisition through localized feature filtering.

The rest of this paper is organized as follows. Section 2 introduces relevant
definitions and background. Then, we explore six critical tasks in molecular
modeling and design. Section 3 presents our benchmark and insights. Section 4
discusses key results and limitations, and Section 5 concludes with a summary
of our contributions and future research directions.

##  2 Methodology

In this section, we present essential background information. We start by
defining different molecular representation forms and vital data sources, then
move to discuss LLMs, highlighting their significance in this field.

###  2.1 Molecular Representation

Molecular structures can be represented in multiple ways for computational
analyses, each with advantages and limitations. These include graph
structures, images, as well as one-dimensional notations. Figure 1. a
showcases examples of various data representation forms, while Figure 1. b
categorizes these representations into molecular internal and external
information. Additionally, it illustrates the relationships between these
forms of representation and their corresponding tasks.

In terms of molecular internal information, SMILES (Simplified Molecular Input
Line Entry System) [15] offers a widely-used one-dimensional notation for
describing molecular structures. InChI (International Chemical Identifier)
[16] provides a unique identifier for chemical substances, facilitating the
standard encoding of molecular information. Further advancing molecular
representations, SELFIES (Self-Referencing Embedded Strings) [17] offers
machine learning-oriented approaches, addressing the limitations of SMILES.
Moreover, graph structures naturally represent molecules, with atoms as nodes
and bonds as edges, effectively capturing both local and global topological
traits.

For external molecular information, images offer a visual 2D or 3D
representation of molecules. IUPAC (International Union of Pure and Applied
Chemistry) [18] names provide a standardized naming system for organic
compounds. Additionally, molecular properties serve as a form of
representation aiding human comprehension. Furthermore, the molecular captions
offer rich contextual information, encompassing textual descriptions of
functions, properties, and inter-molecular relationships.

###  2.2 The Databases and Datasets for Molecular Science

In this section, we highlight vital data sources essential for molecular
science research. As detailed in Table 1, PubChem and ZINC databases provide
extensive compounds for virtual screening. ChEMBL offers comprehensive data on
bioactive molecules, while ChEBI focuses on cataloging small chemical
compounds. DrugBank serves as a valuable resource for drug and target
information. Lastly, the USPTO database, encompassing patent filings and
research data, forms a vital part of the scientific literature.

Table 1: Overview of Data Sources for Molecular Science

Type | Data Source | Content | No. of Entries | Description
---|---|---|---|---
Database |  PubChem [19] | compounds and substances | 116M | molecule database
ZINC [20] | compounds and substances | 230M | virtual screening database
ChEMBL [21] | compounds with text descriptions | 2.4M | bioactive molecule database
ChEBI [22] | small chemical compounds | 61,032 | molecular entity dictionary
DrugBank [23] | small molecule drugs | 12,695 | drug/target database
USPTO | patent filings and data | —— | scientific corpus
Dataset |  C4 [24] | Clean English text | —— | Pretrain
MoMu dataset [25] | Graph-document pairs | 15,613 | Pretrain
MOSES [26] | Molecular structures | 1.9M | Generation
PCdes [27] | SMILES-text descriptions | 1.5K | Molecular retrieval
CheBI-20 [28] | SMILES-text descriptions | 33,010 | Generation and captioning
PubChemSTM [29] | SMILES-text descriptions | 281K | Generation
Mol-Instructions [14] | SMILES-text descriptions | 333K | Generation and captioning
MoleculeNet [30] | Molecular properties. | 700K | Property prediction
USPTO(OSRA) [31] | Images and ground truth | 5,719 | Image recognition
CLEF [32] | Images and ground truth | 865 | Image recognition
JPO [31] | Images and ground truth | 450 | Image recognition
UOB [33] | Images and ground truth | 5,740 | Image recognition
USPTO-50k [34] | Chemical reactions | 50K | Reactions prediction

Moreover, various vital datasets support a wide range of research, from
pretraining to specific molecular science tasks. As shown in Table 1, the
Colossal Clean Crawled Corpus (C4) dataset offers a vast amount of clean
English text scraped from the web for the pretraining of the T5 model. MOSES
provides molecular structures from ZINC, while MoleculeNet encompasses a broad
spectrum of molecular property predictions ranging from quantum mechanics to
biophysics and physiology. Additionally, datasets such as USPTO, CLEF, JPO,
and UOB offer molecular images with ground truth, supporting image recognition
tasks. The PCdes, CheBI-20, and PubChemSTM datasets, along with Mol-
Instructions, provide essential SMILES-text pairs that facilitate cross-
information retrieval, molecule generation, and molecular captioning tasks.

###  2.3 Large Language Models

Transformers, known for their self-attention mechanisms, enable the model to
weigh the importance of different parts of the input data, which is crucial
for understanding complex molecular structures of text modality. LLMs are
Transformer-based models scaled to a large extent.

Among the notable backbone models in this domain are GPT [9] (Generative Pre-
trained Transformer) and BERT [35] (Bidirectional Encoder Representations from
Transformers). Especially in the fine-tuning stage, these models exhibit
remarkable versatility and efficiency, requiring minimal task-specific
adjustments for applications in molecular modeling.

  * •

Encoder-based models: BERT and its derivatives like RoBERTa [36] and domain-
specific variants such as SciBERT [37] and ChemBERTa [38] have been
effectively employed for molecular property prediction.

  * •

Decoder-based models: GPT-2 [9] and ChatGPT [39] are superior in text
generation capabilities. Specific-domain models like MolXPT [40], based on
GPT-2, and MolReGPT [41], based on ChatGPT, showcase strong performance in
generating molecular sequences and descriptions.

  * •

Encoder-Decoder Models: The Transformer architecture itself exemplifies an
end-to-end model, with T5 [24] and BART [42] models further enhancing robust
end-to-end text generation capabilities, advancing domain-specific
applications. For instance, MolT5 [43] is tailored explicitly for molecule
generation and description, while Text+Chem T5 [44] is adept at reaction
prediction. Beyond these, Encoder-Decoder models also encompass multi-modal
models. Prominent examples include BLIP2 [45] and Mini-GPT4 [46], based on
large language model frameworks, and GIT-Mol [47], a multi-modal model capable
of processing image, graph, and text data, demonstrating the versatility and
adaptability of these models in handling diverse data types.

###  2.4 Language Modeling Tasks and Models in Molecular Science

In this section, we present a timeline showcasing the pivotal developments in
transformer-based models for molecular modeling and design, alongside
classifications and architectures, as illustrated in Figure 2. The Details of
the models are shown in the Appendix. A Table 2.

\begin{overpic}[width=433.62pt]{figure2_timeline.pdf}
\put(26.0,85.5){\cite[cite]{[\@@bibref{}{seidl2023enhancing}{}{}]}}
\put(6.0,75.0){\cite[cite]{[\@@bibref{}{zhu2023dual}{}{}]}}
\put(38.0,80.0){\cite[cite]{[\@@bibref{}{ross2022large}{}{}]}}
\put(38.0,77.8){\cite[cite]{[\@@bibref{}{guo2022multilingual}{}{}]}}
\put(57.0,86.5){\cite[cite]{[\@@bibref{}{jiang2023noisemol}{}{}]}}
\put(57.0,84.3){\cite[cite]{[\@@bibref{}{gao2023transfoxmol}{}{}]}}
\put(40.0,135.0){\cite[cite]{[\@@bibref{}{ying2021transformers}{}{}]}}
\put(9.0,82.0){\cite[cite]{[\@@bibref{}{xie2023self}{}{}]}}
\put(55.0,91.0){\cite[cite]{[\@@bibref{}{mazuz2023molecule}{}{}]}}
\put(26.0,88.5){\cite[cite]{[\@@bibref{}{fang2023domainagnostic}{}{}]}}
\put(46.5,80.0){\cite[cite]{[\@@bibref{}{rothchild2021c5t5}{}{}]} }
\put(35.5,75.5){\cite[cite]{[\@@bibref{}{Lu_Zhang_2022}{}{}]} }
\put(35.0,95.5){\cite[cite]{[\@@bibref{}{livne2023nach0}{}{}]} }
\put(57.5,77.3){\cite[cite]{[\@@bibref{}{he2022transformer}{}{}]}}
\put(57.0,88.6){\cite[cite]{[\@@bibref{}{bran2023chemcrow}{}{}]}}
\put(9.0,84.3){\cite[cite]{[\@@bibref{}{tang2023mollm}{}{}]}} \end{overpic}
Figure 2: An overview of model developments and architectures. a. Timeline of
the key developments of LLMs, with models based on the Transformer
architecture differentiated by various colors to denote the backbone. b. Tasks
and models, clarifies the relationship between six downstream tasks and model
architectures. c. Encoder-Decoder model Architectures, delineating three main
frameworks: (1) Text-Text primarily focused on text translation tasks; (2)
Graph-Text, predominantly used in contrastive learning frameworks and serving
as an encoder for downstream tasks; (3) Image-Text, chiefly applied in
Molecular Image Recognition tasks.

#### Molecule captioning

It aims to develop textual descriptions based on given molecular
representations. It involves not only recognizing the molecular structure but
also interpreting and conveying its functional properties, potential uses, and
any relevant scientific context.

MolT5 [43] is a groundbreaking model that uses the T5 architecture for end-to-
end text tasks, achieving exceptional performance in single-modality models.
Following this, MolXPT [40], built on GPT-2, utilizes a large corpus of PubMed
titles and abstracts along with SMILES sequences from PubChem [19]. It
performs exceptionally well in few-shot learning and generative pre-training.
As for multi-modality models, MoMu [25] and MolFM [64] employ cross-modal
contrastive learning techniques. They fuse graphical and textual data through
their encoders and use MolT5 as a decoder. MolCA [65] combines Q-Former with
contrastive learning to integrate SMILES strings and graphs, leveraging
Galactica’s [66] generative capabilities for captioning. In a similar vein,
GIT-Mol [47] advances this integration with its more versatile GIT-Former
architecture, processing text, graphs, and images.

#### IUPAC Name Recognition

Unlike free-form textual descriptions, IUPAC names possess the advantage of
uniformity and specificity, making them highly structured and predictable.
Furthermore, the task of IUPAC name recognition involves not just the decoding
of these structured names but also understanding the underlying molecular
structure they represent. Struct2IUPAC [67] employs a Transformer-based
architecture, complemented by a specialized tokenizer adept at processing both
SMILES strings and IUPAC nomenclature. Moreover, the TransAntivirus [68] goes
a step further by adopting the versatile T5 model architecture. This approach
not only enables conversions but also facilitates the evaluation of generated
molecules on multiple metrics, including uniqueness, validity, and a range of
chemical properties.

#### Molecular Property Prediction

Molecular representation is critical in molecular modeling, providing ways to
describe and encode molecular structures for computational use. Within this
domain, molecular property prediction is essential, assessing attributes like
solubility, toxicity, and protein binding affinity. Based on BERT, KV-PLM [27]
employs a dual tokenizer to better recognize specific atoms and functional
groups within SMILES strings. LLM4Mol [69] leverages the power of LLMs to
introduce a novel molecular representation through text-based explanations. As
for multi-modal methods, MoMu [25] applies contrastive learning techniques to
assimilate modality-specific and complementary information from text and graph
data. Post-encoding, embeddings are transformed into fixed-length vectors via
a pooling layer, employing methods like average pooling and max pooling.
Average pooling calculates the mean of all sequence vectors for a general
context representation. Max pooling selects each feature’s maximum value
across the sequence, highlighting key molecular features.

#### Molecular Retrieval

Molecular retrieval aims for efficient, accurate identification of molecules
in large datasets. Its effectiveness relies on the type of molecular data
used, like structural formulas or SMILES strings. In cross-modal retrieval
tasks, Text2Mol [28] excels by combining natural language with molecular data
into a unified semantic space for effective retrieval. Similarly, MoMu [25]
and MolFM [64] use advanced contrastive learning to merge text and molecular
graphics, enhancing the feature space for robust retrieval tasks. As for the
pooling layers, similar to those detailed in the molecular property prediction
section. This process ensures that the final vectors retain the most salient
features from both textual and graphical inputs, enabling more accurate and
efficient molecular retrieval.

#### Molecular Image Recognition

In our research, molecular image recognition focuses on converting visual
representations of molecular structures into standardized, textual formats
like SMILES. This task is particularly crucial for automating the
interpretation of chemical literature, where molecular images are frequently
used to describe compounds. DECIMER 1.0 [70], Image2SMILES [71] and MICER [72]
employ Transformer-based architectures and image augmentation, handling large
datasets effectively. SwinOCSR [73] utilizes Swin Transformers [74] for global
feature extraction and tackles token imbalance with focal loss, streamlining
DeepSMILES generation. Additionally, MolScribe [75] shows strong performance
against low-quality images and includes confidence assessment for its
predictions, enhancing output reliability.

#### Molecule Generation

Our research focuses on developing one-dimensional textual representations,
particularly in SMILES format. This includes text-based de novo molecule
generation and molecular optimization, due to their methodological
similarities. In molecular generation, models adept at molecule captioning are
equally capable of molecule generation. MolT5 [43] marks a significant
advancement in this field, surpassing traditional RNNs and Transformers with
its efficiency. BioT5 [76] further expands this to include protein-related
texts, focusing on SELFIES and extending to Protein Property Prediction and
Drug-target Interaction.

Molecular optimization aims to refine molecules for improved properties,
aligning with generative goals. MolGPT [77] utilizes the GPT architecture’s
self-attention and conditional generation for effective molecule control and
property-based generation. MoleculeSTM [29] employs MegaMolBART[78] and
contrastive learning on graphs and text, optimizing target attributes like
binding affinity and drug-relatedness with precision. ChatMol [79] takes an
interactive approach, combining empirical data, spatial knowledge, and
language associations for enhanced molecular design.

##  3 Evaluation and Results

In this chapter, we propose ChEBI-20-MM, a comprehensive benchmark developed
from the ChEBI-20 dataset, integrating multi-modal data such as InChI, IUPAC,
SELFIES, and images. This benchmark is designed to assess the chemical
knowledge of models across tasks such as molecule generation, image and IUPAC
recognition, molecular captioning, and retrieval, featuring in-depth analysis
of the compatibility between modalities and models, as well as an exploration
into models’ tendencies for knowledge acquisition.

###  3.1 ChEBI-20-MM and Experimental Setup

In molecular representations, molecular text descriptions, typically scarce,
now see increased utilization with MolT5’s ChEBI-20 dataset in molecule
generation and captioning. To facilitate multi-modal data evaluation, we
expand our data modality variety, introducing ChEBI-20-MM. This benchmark
effectively evaluates the versatility of data types and model combinations in
describing, generating, and embedding molecular data. Figure 3 showcases our
evaluation of prevalent backbone models and experiments. For comparing models’
embedding capabilities, our experiments include datasets from MoleculeNet,
enabling thorough assessments in biochemical and physicochemical contexts.

We conduct 1263 experiments, encompassing 291 generative and retrieval tasks
from the ChEBI-20-MM benchmark and 972 property prediction experiments. For
text-to-text tasks, our models include 11 architectural variations, combining
three decoder-only models, four encoder-decoder models, and four composite
models with MolT5 as the decoder paired with various encoders. We conduct
experiments based on the setup, performing each experiment three times to
calculate the mean and standard deviation of the results. Then, we analyze the
suitability of data sources for language models and chemical space coverage,
as shown in Appendix. B Figure 6. The description of the metrics is also
located in Appendix B.

Figure 3: Benchmark Experiments Overview. Our study encompasses tests across
eight primary model architectures, each featuring 3-4 common backbone models
or composite models within its category. The figure annotates the base model
parameter size for clarity. In total, 1263 experimental setups were conducted,
demonstrating the adaptability of various model architectures to different
task types.

###  3.2 Experimental Results

Figure 4: Results of Benchmark. a. Modal Transition Probability Matrix. This
matrix presents the performance in text generation and property prediction
tasks. The vertical axis represents input modalities, while the horizontal
axis denotes output modalities. b. Encoders and Decoders in Nine Text-to-Text
Tasks. This illustration highlights the frequency of various models appearing
in the top 5 rankings. The T5 series models exhibit a dominant presence. c.
Encoders, Pooling Mechanisms, and Retrieval Performance in Embedding Tasks.
Alongside model rankings, the figure indicates that average pooling is a
preferred choice for the pooling layer. Additionally, from a molecular
retrieval perspective, SMILES-IUPAC retrieval demonstrates the highest
efficiency.

For the analysis of modality adaptation to different tasks, as depicted in
Figure 4. a. We select representative metrics indicative of successful task
completion, such as the METEOR score for molecule captioning and the ROC_AUC
for property classification tasks. We fill out the modal transition
probability matrix, which provides insights into the most suitable data
modalities for different task types. Regarding model analysis for different
tasks, Figure 4(b) and (c) demonstrate the frequency of appearances of
encoders and decoders in nine text-to-text tasks, as well as eight encoders
and two pooling mechanisms in embedding tasks. Detailed experimental results
are shown in Appendix. C Table 3-Table 7. From the results, we present the
following modality insights:

  * •

IUPAC Names for Generation and Captioning: IUPAC names are conducive to text-
based molecule generation and molecule captioning.

  * •

SMILES for IUPAC Recognition:  SMILES representations are particularly well-
suited for IUPAC recognition compared to other molecular internal information.

  * •

Preferred Modalities for Captioning:  Since some captions include the IUPAC
name of the molecule, IUPAC is the most appropriate modality for molecule
captioning, followed by SMILES.

  * •

Choosing Modalities for Retrieval:  For molecule retrieval tasks targeting
captions, IUPAC is the optimal modality, followed by SMILES. Conversely, when
the target is IUPAC names, SMILES becomes the most favorable modality.

  * •

Graph vs. Textual Modalities in Property Prediction:  In molecular property
prediction tasks, among the selected top five rank combinations from 90
groups, graph modalities appear 40 times and SMILES 25 times, indicating a
clear advantage for graph modalities and the relative superiority of SMILES
over other textual representations.

As for the model performance insights:

  * •

T5 Series Excellence: The T5 series models show excellent performance across
these tasks.

  * •

Graph Models vs. T5 Series:  Only the significantly larger T5 series models
surpass GNNs, affirming the superiority of smaller-scale graph models for
molecular embedding.

  * •

Efficacy of Average Pooling:

The average pooling mechanism tends to achieve superior performance more
easily.

###  3.3 Analysis of Model Knowledge-Learning Preferences

In exploring the knowledge-learning preferences of models in the process of
molecular SLM, this section introduces a language-driven, statistically
significant, interpretable method. This tool constructs a token mapping matrix
between input data and inference outcomes, revealing the model’s mechanism. We
typically view high-frequency token mappings as indicative of specialized
knowledge. Our approach identifies two types of high-frequency token mappings:
general pairs, which are widespread and lack specific context, and particular
pairs, essential for certain contexts but not always the most frequent. The
challenge arises when standard analysis methods prioritize these general high-
frequency mappings, consequently overlooking the specific ones that often
carry more valuable contextual insights.

We utilize a localized feature filtering method for high-frequency selection
to eliminate common high-frequency mapping pairs, thus more accurately
filtering the model’s knowledge mappings. Subsequently, a significant
statistical method is applied to determine the filtering threshold,
identifying high-frequency mapping pairs that reflect the model’s chemical
knowledge.

Proof Point 1: Sorting the matrix $A$ in descending order by row and column
totals enhances the similarity among adjacent values. Assuming each $A_{ij}$
follows an independent and identical distribution, such as normal
distribution, the row sums $R_{i}$ and column sums $C_{j}$ mimic this
distribution. Consequently, in the sorted matrix $A^{\prime}$, elements next
to each other are more likely to come from rows or columns with similar
totals.

Considering $A^{\prime}_{kl}$ and its neighbors $A^{\prime}_{k\pm 1,l\pm 1}$
in $A^{\prime}$, which were close in the sum in the original matrix $A$, the
probability they are similar is mathematically represented as:

$P(A^{\prime}_{kl}\approx A^{\prime}_{k\pm 1,l\pm 1}|R_{i}\approx R_{i\pm
1},C_{j}\approx C_{j\pm 1})>P(A_{ij}\approx A_{i\pm 1,j\pm 1})$

This highlights that adjacent elements in $A^{\prime}$ are more likely to have
close values compared to those in $A$, due to the sorting based on similar row
and column sums.

Proof Point 2: To uncover significant patterns within a matrix, we introduce a
threshold $T$, aiming to spotlight elements $A_{ij}$ that significantly
surpass the mean plus a certain number of standard deviations of their
neighboring values. This process starts with assuming elements in matrix $A$
are drawn from a distribution, enabling the calculation of each element’s
neighborhood mean $\mu_{neighbor}$ and standard deviation $\sigma_{neighbor}$.
This step evaluates whether an element exceeds the set threshold, the
condition to filter elements that starkly differ from their surrounding
context:

$A_{ij}>\mu_{neighbor}+T\cdot\sigma_{neighbor}$

To assess the actual significance of these findings, we calculate the actual
proportion $P_{actual}$ of elements exceeding this threshold across the matrix
and compare it to the expected proportion $P_{expected}$ under a normal
distribution. The actual and expected proportion are determined by the
fraction of elements satisfying our condition:

$P_{actual}=\frac{\sum I_{ij}}{n\times m}$

$P_{expected}=1-\Phi\left(\frac{\mu_{neighbor}+T\cdot\sigma_{neighbor}-\mu}{\sigma}\right)$

where $I_{ij}$ is an indicator function that equals 1 when the condition is
met and 0 otherwise, and $n\times m$ represents the matrix size. The $\Phi$
represents the cumulative distribution function(CDF) of the standard normal
distribution, and $\mu$ and $\sigma$ are the mean and standard deviation of
the entire matrix’s elements.

Finally, a statistical test, such as the Z-test, evaluates the significance of
the observed proportion against the expected, indicating whether the
identified patterns are indeed statistically significant beyond random chance:

$Z=\frac{P_{actual}-P_{expected}}{\sqrt{\frac{P_{expected}(1-P_{expected})}{n\times
m}}}$

This succinct methodology not only clarifies the process of identifying
statistically significant patterns but also underlines its effectiveness in
distinguishing genuine high-frequency elements from those of a random
distribution, offering valuable insights into the structure of the knowledge-
learning preferences of models.

###  3.4 Case Studies on Model Knowledge-Learning Preferences

Our analysis aims to discern model preferences for knowledge acquisition. To
enhance interpretability, we focus on the IUPAC-to-caption mapping process,
which is closely aligned with natural language tasks. Figure 5. a displays the
token mapping matrix after normalization and rearrangement based on the total
counts of rows and columns, selecting the top 20 high-frequency chemical
tokens in IUPAC names and captions. General high-frequency mappings, such as
oxy, methyl, and hydroxy, appear extensively across various tokens.
Traditional filtering methods might capture many of these mappings yet
potentially exclude those with specific contextual significance.

Figure 5:  Co-occurrence Patterns and Insights. a. IUPAC-to-Caption Tokens
Mapping Matrix of T5 The matrix displays the y-axis as high-frequency tokens
from IUPAC and the x-axis as chemical tokens selected from the captions. This
visualization facilitates the identification of data points with high co-
occurrence frequencies. b. Threshold T Analysis. There are 100 thresholds
within the range [0,3] for $T$. As the threshold increases, the criteria for
selecting specific high-frequency pairs become more stringent, reducing their
number while also affecting significance levels. c. Examples of T5 Knowledge-
Learning Preferences. In our IUPAC-to-caption analysis, we select four mapping
patterns that reveal the model’s grasp of chemical structures and language,
aligning with known chemical knowledge. (1) The mapping from ‘ent’ to ‘methyl’
demonstrates the model’s ability to identify molecular structural elements,
like the carbonyl group in Pentan-2-one. (2) The ‘cycl’ to ‘hydroxy’ and ‘one’
mapping indicates the model’s understanding of complex structures, recognizing
the cyclohexanol structure and hydroxyl groups in
3,3,5-Trimethylcyclohexan-1-ol. (3) Mapping ‘ace’ to ‘amino’ shows the model’s
knowledge of biochemical principles, identifying the role of amino groups in
2-aminoacetate. (4) The ‘lan’ and ‘phospho’ to ‘phosphat’ mapping reveals the
model’s insight into phosphate groups and their biological importance.

To find specific high-frequency mapping pairs, as shown in Figure 5. b, at
$T=2.788$, the Z-test reaches its maximum value of 2.405, corresponding to a
confidence level of 98.2%. After identifying 21 specific token mapping pairs
and removing those with identical row and column names, we are left with 16
unique pairs. By consolidating mappings with the same leading or trailing
tokens, we obtain the following 9 groups of token mappings:

  * •

[‘ent’] $\rightarrow$ [‘methyl’]

  * •

[‘lan’, ‘phospho’] $\rightarrow$ [‘phosphat’]

  * •

[‘ace’] $\rightarrow$ [‘amino’]

  * •

[‘ryl’] $\rightarrow$ [‘acy’, ‘ate’, ‘acid’, ‘conjug’]

  * •

[‘hydr’] $\rightarrow$ [‘one’]

  * •

[‘cycl’] $\rightarrow$ [‘hydroxy’, ‘one’]

  * •

[‘phen’] $\rightarrow$ [‘group’]

  * •

[‘min’] $\rightarrow$ [‘ine’]

  * •

[‘hydroxy’] $\rightarrow$ [‘acid’, ‘mono’]

These mapping patterns illustrate the diversity and specificity of chemical
token relationships captured through our analysis. Following these mapping
patterns, we randomly select corresponding molecular IUPAC names and their
captions, as shown in Figure 5. c.

The T5 model exhibits a robust understanding of molecules featuring functional
groups such as carbonyls (e.g., ketones), oxygen-containing groups (e.g.,
alcohols), nitrogen-containing groups (e.g., amines), and phosphate groups,
accurately interpreting their chemical properties and impacts on molecular
characteristics. Through a series of model knowledge-learning insights, we
demonstrate that the observed mapping patterns are not merely coincidental but
reflective of the model’s inherent ability to process and interpret complex
chemical information.

##  4 Discussion

#### The Impact of Data Modalities

To identify how data modalities influence model performance in molecular
science tasks, we develop a multi-modal benchmark and carry out thousands of
experiments. Yet, our focus on prevalent data modalities and established
backbone models may overlook some Fingerprint modalities and innovative
training strategies. Therefore, our findings provide comparative insights,
highlighting areas for more profound research in future work.

#### Scientific Insights Discovery

To explore the knowledge-learning preferences, we provide an interpretable
analysis, revealing the foundational mechanisms in molecular SLM. It confirms
that models are capable of modeling chemical knowledge. Nevertheless, our
investigation primarily adopts an NLP standpoint, with limited incorporation
of scientific principles. To more clearly showcase the models’ capabilities in
scientific understanding, future research should incorporate a broader
spectrum of biochemical and physical chemistry knowledge.

#### Benchmarking of SLM

While we strive to include a broad range of metrics for various tasks, our
approach to molecular generation tasks primarily relies on similarity metrics
in comparison to target texts. This methodology falls short in evaluating the
actual volume of knowledge acquired by the models. A more comprehensive
assessment framework is needed to accurately gauge the learning capabilities
of these models in molecular generation.

##  5 Conclusion

Our study presents a comprehensive review of LLMs in molecular modeling and
design. We propose a multi-modal benchmark named ChEBI-20-MM, substantiated by
1263 experiments to evaluate LLMs, with a focus on the model’s adaptability of
data modalities and knowledge acquisition. We demonstrate insights into the
most suitable modalities for tasks by the modal transition probability matrix.
Furthermore, we utilize statistically localized feature filtering to discover
model knowledge-learning preferences. These advancements lay the groundwork
for future progress in molecular SLM.

In future research, we will focus on a sophisticated exploration of multi-
modal data fusion to understand how various data modalities enhance model
performance. We plan to tailor evaluation metrics specifically for different
SLM tasks, moving beyond merely focusing on similarity to the target text.
Additionally, integrating scientific perspectives, including biochemical and
pharmacological analyses, will offer a more comprehensive understanding of the
models’ capabilities. These steps will bridge the gap between computational
modeling and practical scientific applications, unlocking new potential in
molecular sciences.

## Data and Software Availability:

The data and software can be accessed at https://github.com/AI-HPC-Research-
Team/SLM4Mol. It is available for non-commercial use.

## Funding

This work is supported by grants from the National Natural Science Foundation
of China (61902446, 62172456, and 91937302) and the Peng Cheng Cloud-Brain of
Peng Cheng Laboratory.

## CRediT authorship contribution statement

Pengfei Liu: Conceptualization, Methodology, Data curation, Model traning,
Writing-original draft. Jun Tao: Writing review editing. Zhixiang Ren:
Conceptualization, Formal analysis, Supervision, Funding acquisition, Writing
review editing.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Acknowledgment

This research was supported by the Peng Cheng Laboratory and Peng Cheng Cloud-
Brain.

## References

  * [1] K. Heikamp and J. Bajorath, “Support vector machines for drug discovery,” Expert opinion on drug discovery, vol. 9, no. 1, pp. 93–104, 2014.
  * [2] O. Wieder, S. Kohlbacher, M. Kuenemann, A. Garon, P. Ducrot, T. Seidel, and T. Langer, “A compact review of molecular property prediction with graph neural networks,” Drug Discovery Today: Technologies, vol. 37, pp. 1–12, 2020.
  * [3] S. Liu, H. Wang, W. Liu, J. Lasenby, H. Guo, and J. Tang, “Pre-training molecular graph representation with 3d geometry,” in International Conference on Learning Representations, 2021.
  * [4] G. Zhou, Z. Gao, Q. Ding, H. Zheng, H. Xu, Z. Wei, L. Zhang, and G. Ke, “Uni-mol: A universal 3d molecular representation learning framework,” in The Eleventh International Conference on Learning Representations, 2022.
  * [5] A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, “Generative adversarial networks: An overview,” IEEE signal processing magazine, vol. 35, no. 1, pp. 53–65, 2018.
  * [6] N. De Cao and T. Kipf, “Molgan: An implicit generative model for small molecular graphs.,” in ICML 2018 workshop on Theoretical Foundations and Applications of Deep Generative Models., 2018.
  * [7] M. Xu, L. Yu, Y. Song, C. Shi, S. Ermon, and J. Tang, “Geodiff: A geometric diffusion model for molecular conformation generation,” in International Conference on Learning Representations, 2021.
  * [8] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  * [9] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., “Language models are unsupervised multitask learners,” OpenAI blog, 2019.
  * [10] OpenAI, “GPT-4 technical report,” CoRR, vol. abs/2303.08774, 2023.
  * [11] Y. Du, T. Fu, J. Sun, and S. Liu, “Molgensurvey: A systematic survey in machine learning models for molecule design,” arXiv preprint arXiv:2203.14500, 2022.
  * [12] Y. Fang, Q. Zhang, Z. Chen, X. Fan, and H. Chen, “Knowledge-informed molecular learning: A survey on paradigm transfer,” arXiv preprint arXiv:2202.10587, 2022.
  * [13] T. Guo, K. Guo, Z. Liang, Z. Guo, N. V. Chawla, O. Wiest, X. Zhang, et al., “What indeed can gpt models do in chemistry? a comprehensive benchmark on eight tasks,” arXiv preprint arXiv:2305.18365, 2023.
  * [14] Y. Fang, X. Liang, N. Zhang, K. Liu, R. Huang, Z. Chen, X. Fan, and H. Chen, “Mol-instructions: A large-scale biomolecular instruction dataset for large language models,” arXiv preprint arXiv:2306.08018, 2023.
  * [15] D. Weininger, “Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules,” Journal of chemical information and computer sciences, vol. 28, no. 1, pp. 31–36, 1988.
  * [16] S. Heller, A. McNaught, S. Stein, D. Tchekhovskoi, and I. Pletnev, “Inchi-the worldwide chemical structure identifier standard,” Journal of cheminformatics, vol. 5, no. 1, pp. 1–9, 2013.
  * [17] M. Krenn, F. Häse, A. Nigam, P. Friederich, and A. Aspuru-Guzik, “Self-referencing embedded strings (selfies): A 100% robust molecular string representation,” Machine Learning: Science and Technology, vol. 1, no. 4, p. 045024, 2020.
  * [18] G. L. Long and J. D. Winefordner, “Limit of detection. a closer look at the iupac definition,” Analytical chemistry, vol. 55, no. 7, pp. 712A–724A, 1983.
  * [19] S. Kim, J. Chen, T. Cheng, A. Gindulyte, J. He, S. He, Q. Li, B. A. Shoemaker, P. A. Thiessen, B. Yu, et al., “Pubchem 2023 update,” Nucleic acids research, vol. 51, no. D1, pp. D1373–D1380, 2023.
  * [20] J. J. Irwin, K. G. Tang, J. Young, C. Dandarchuluun, B. R. Wong, M. Khurelbaatar, Y. S. Moroz, J. Mayfield, and R. A. Sayle, “Zinc20—a free ultralarge-scale chemical database for ligand discovery,” Journal of chemical information and modeling, vol. 60, no. 12, pp. 6065–6073, 2020.
  * [21] B. Zdrazil, E. Felix, F. Hunter, E. J. Manners, J. Blackshaw, S. Corbett, M. de Veij, H. Ioannidis, D. Mendez Lopez, J. F. Mosquera, et al., “The chembl database in 2023: a drug discovery platform spanning multiple bioactivity data types and time periods,” Nucleic Acids Research, p. gkad1004, 2023.
  * [22] J. Hastings, G. Owen, A. Dekker, M. Ennis, N. Kale, V. Muthukrishnan, S. Turner, N. Swainston, P. Mendes, and C. Steinbeck, “Chebi in 2016: Improved services and an expanding collection of metabolites,” Nucleic acids research, vol. 44, no. D1, pp. D1214–D1219, 2016.
  * [23] D. S. Wishart, Y. D. Feunang, A. C. Guo, E. J. Lo, A. Marcu, J. R. Grant, T. Sajed, D. Johnson, C. Li, Z. Sayeeda, et al., “Drugbank 5.0: a major update to the drugbank database for 2018,” Nucleic acids research, vol. 46, no. D1, pp. D1074–D1082, 2018.
  * [24] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485–5551, 2020.
  * [25] B. Su, D. Du, Z. Yang, Y. Zhou, J. Li, A. Rao, H. Sun, Z. Lu, and J.-R. Wen, “A molecular multimodal foundation model associating molecule graphs with natural language,” arXiv preprint arXiv:2209.05481, 2022.
  * [26] D. Polykovskiy, A. Zhebrak, B. Sanchez-Lengeling, S. Golovanov, O. Tatanov, S. Belyaev, R. Kurbanov, A. Artamonov, V. Aladinskiy, M. Veselov, et al., “Molecular sets (moses): a benchmarking platform for molecular generation models,” Frontiers in pharmacology, vol. 11, p. 565644, 2020.
  * [27] Z. Zeng, Y. Yao, Z. Liu, and M. Sun, “A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals,” Nature communications, vol. 13, no. 1, p. 862, 2022.
  * [28] C. Edwards, C. Zhai, and H. Ji, “Text2mol: Cross-modal molecule retrieval with natural language queries,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 595–607, 2021.
  * [29] S. Liu, W. Nie, C. Wang, J. Lu, Z. Qiao, L. Liu, J. Tang, C. Xiao, and A. Anandkumar, “Multi-modal molecule structure-text model for text-based retrieval and editing,” arXiv preprint arXiv:2212.10789, 2022.
  * [30] Z. Wu, B. Ramsundar, E. N. Feinberg, J. Gomes, C. Geniesse, A. S. Pappu, K. Leswing, and V. Pande, “Moleculenet: a benchmark for molecular machine learning,” Chemical science, vol. 9, no. 2, pp. 513–530, 2018.
  * [31] “Osra validation datasets.” https://sourceforge.net/p/osra/wiki/Validation/, 2020.  Accessed: 2020-06-24.
  * [32] “CLEF-IP 2012 Structure Recognition Test Set.” https://www.ifs.tuwien.ac.at/~clef-ip/download/2012/qrels/clef-ip-2012-chem-recognition-qrels.tgz, 2012.  Accessed: 2020-06-29.
  * [33] “Molrecuob benchmark dataset.” https://www.cs.bham.ac.uk/research/groupings/reasoning/sdag/chemical.php, 2020.  Accessed: 2020-06-29.
  * [34] N. Schneider, N. Stiefl, and G. A. Landrum, “What’s what: The (nearly) definitive guide to reaction role assignment,” Journal of chemical information and modeling, vol. 56, no. 12, pp. 2336–2346, 2016.
  * [35] J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of NAACL-HLT, pp. 4171–4186, 2019.
  * [36] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019.
  * [37] I. Beltagy, K. Lo, and A. Cohan, “Scibert: A pretrained language model for scientific text,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3615–3620, 2019.
  * [38] S. Chithrananda, G. Grand, and B. Ramsundar, “Chemberta: large-scale self-supervised pretraining for molecular property prediction,” arXiv preprint arXiv:2010.09885, 2020.
  * [39] J. Schulman, B. Zoph, C. Kim, J. Hilton, J. Menick, J. Weng, J. F. C. Uribe, L. Fedus, L. Metz, M. Pokorny, et al., “Chatgpt: Optimizing language models for dialogue,” OpenAI blog, 2022.
  * [40] Z. Liu, W. Zhang, Y. Xia, L. Wu, S. Xie, T. Qin, M. Zhang, and T.-Y. Liu, “Molxpt: Wrapping molecules with text for generative pre-training,” arXiv preprint arXiv:2305.10688, 2023.
  * [41] J. Li, Y. Liu, W. Fan, X.-Y. Wei, H. Liu, J. Tang, and Q. Li, “Empowering molecule discovery for molecule-caption translation with large language models: A chatgpt perspective,” arXiv preprint arXiv:2306.06615, 2023.
  * [42] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871–7880, 2020.
  * [43] C. Edwards, T. Lai, K. Ros, G. Honke, K. Cho, and H. Ji, “Translation between molecules and natural language,” in 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, 2022.
  * [44] D. Christofidellis, G. Giannone, J. Born, O. Winther, T. Laino, and M. Manica, “Unifying molecular and textual representations via multi-task language modelling,” arXiv preprint arXiv:2301.12586, 2023.
  * [45] J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” arXiv preprint arXiv:2301.12597, 2023.
  * [46] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” arXiv preprint arXiv:2304.10592, 2023.
  * [47] P. Liu, Y. Ren, and Z. Ren, “Git-mol: A multi-modal large language model for molecular science with graph, image, and text,” arXiv preprint arXiv:2308.06911, 2023.
  * [48] P. Seidl, A. Vall, S. Hochreiter, and G. Klambauer, “Enhancing activity prediction models in drug discovery with the ability to understand human language,” arXiv preprint arXiv:2303.03363, 2023.
  * [49] J. Zhu, Y. Xia, L. Wu, S. Xie, W. Zhou, T. Qin, H. Li, and T.-Y. Liu, “Dual-view molecular pre-training,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3615–3627, 2023.
  * [50] J. Ross, B. Belgodere, V. Chenthamarakshan, I. Padhi, Y. Mroueh, and P. Das, “Large-scale chemical language representations capture molecular structure and properties,” Nature Machine Intelligence, vol. 4, no. 12, pp. 1256–1264, 2022.
  * [51] Z. Guo, P. Sharma, A. Martinez, L. Du, and R. Abraham, “Multilingual molecular representation learning via contrastive pre-training,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3441–3453, 2022.
  * [52] J. Jiang, R. Zhang, Y. Yuan, T. Li, G. Li, Z. Zhao, and Z. Yu, “Noisemol: A noise-robusted data augmentation via perturbing noise for molecular property prediction,” Journal of Molecular Graphics and Modelling, vol. 121, p. 108454, 2023.
  * [53] J. Gao, Z. Shen, Y. Xie, J. Lu, Y. Lu, S. Chen, Q. Bian, Y. Guo, L. Shen, J. Wu, et al., “Transfoxmol: predicting molecular property with focused attention,” Briefings in Bioinformatics, p. bbad306, 2023.
  * [54] C. Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu, “Do transformers really perform badly for graph representation?,” Advances in Neural Information Processing Systems, vol. 34, pp. 28877–28888, 2021.
  * [55] A. Xie, Z. Zhang, J. Guan, and S. Zhou, “Self-supervised learning with chemistry-aware fragmentation for effective molecular property prediction,” Briefings in Bioinformatics, p. bbad296, 2023.
  * [56] E. Mazuz, G. Shtar, B. Shapira, and L. Rokach, “Molecule generation using transformers and policy gradient reinforcement learning,” Scientific Reports, vol. 13, no. 1, p. 8799, 2023.
  * [57] Y. Fang, N. Zhang, Z. Chen, X. Fan, and H. Chen, “Domain-agnostic molecular generation with self-feedback,” 2023.
  * [58] D. Rothchild, A. Tamkin, J. Yu, U. Misra, and J. Gonzalez, “C5t5: Controllable generation of organic molecules with transformers,” arXiv preprint arXiv:2108.10307, 2021.
  * [59] J. Lu and Y. Zhang, “Unified deep learning model for multitask reaction predictions with explanation,” Journal of Chemical Information and Modeling, p. 1376–1387, Mar 2022.
  * [60] M. Livne, Z. Miftahutdinov, E. Tutubalina, M. Kuznetsov, D. Polykovskiy, A. Brundyn, A. Jhunjhunwala, A. Costa, A. Aliper, and A. Zhavoronkov, “nach0: Multimodal natural and chemical languages foundation model,” arXiv preprint arXiv:2311.12410, 2023.
  * [61] J. He, F. Mattsson, M. Forsberg, E. J. Bjerrum, O. Engkvist, C. Tyrchan, W. Czechtizky, et al., “Transformer neural network for structure constrained molecular optimization,” Journal of Cheminformatics, 2022.
  * [62] A. M. Bran, S. Cox, A. D. White, and P. Schwaller, “Chemcrow: Augmenting large-language models with chemistry tools,” arXiv preprint arXiv:2304.05376, 2023.
  * [63] X. Tang, A. Tran, J. Tan, and M. Gerstein, “Mollm: Integrating 3d and 2d molecular representations with biomedical text via a unified pre-trained language model,” bioRxiv, pp. 2023–11, 2023.
  * [64] Y. Luo, K. Yang, M. Hong, X. Liu, and Z. Nie, “Molfm: A multimodal molecular foundation model,” arXiv preprint arXiv:2307.09484, 2023.
  * [65] Z. Liu, S. Li, Y. Luo, H. Fei, Y. Cao, K. Kawaguchi, X. Wang, and T.-S. Chua, “Molca: Molecular graph-language modeling with cross-modal projector and uni-modal adapter,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15623–15638, 2023.
  * [66] R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic, “Galactica: A large language model for science,” arXiv preprint arXiv:2211.09085, 2022.
  * [67] L. Krasnov, I. Khokhlov, M. Fedorov, and S. Sosnin, “Struct2iupac–transformer-based artificial neural network for the conversion between chemical notations,” Scientific Reports, 2021.
  * [68] J. Mao, J. Wang, A. Zeb, K.-H. Cho, H. Jin, J. Kim, O. Lee, Y. Wang, and K. T. No, “Transformer-based molecular generative model for antiviral drug design,” Journal of Chemical Information and Modeling, 2023.
  * [69] C. Qian, H. Tang, Z. Yang, H. Liang, and Y. Liu, “Can large language models empower molecular property prediction?,” arXiv preprint arXiv:2307.07443, 2023.
  * [70] K. Rajan, A. Zielesny, and C. Steinbeck, “Decimer 1.0: deep learning for chemical image recognition using transformers,” Journal of Cheminformatics, vol. 13, no. 1, pp. 1–16, 2021.
  * [71] I. Khokhlov, L. Krasnov, M. V. Fedorov, and S. Sosnin, “Image2smiles: Transformer-based molecular optical recognition engine,” Chemistry-Methods, vol. 2, no. 1, p. e202100069, 2022.
  * [72] J. Yi, C. Wu, X. Zhang, X. Xiao, Y. Qiu, W. Zhao, T. Hou, and D. Cao, “Micer: a pre-trained encoder-decoder architecture for molecular image captioning,” Bioinformatics (Oxford, England), vol. 38, no. 19, pp. 4562–4572, 2022.
  * [73] Z. Xu, J. Li, Z. Yang, S. Li, and H. Li, “Swinocsr: end-to-end optical chemical structure recognition using a swin transformer,” Journal of Cheminformatics, vol. 14, no. 1, pp. 1–13, 2022.
  * [74] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012–10022, 2021.
  * [75] Y. Qian, J. Guo, Z. Tu, Z. Li, C. W. Coley, and R. Barzilay, “Molscribe: Robust molecular structure recognition with image-to-graph generation,” Journal of Chemical Information and Modeling, vol. 63, no. 7, pp. 1925–1934, 2023.
  * [76] Q. Pei, W. Zhang, J. Zhu, K. Wu, K. Gao, L. Wu, Y. Xia, and R. Yan, “Biot5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1102–1123, 2023.
  * [77] V. Bagal, R. Aggarwal, P. Vinod, and U. D. Priyakumar, “Molgpt: molecular generation using a transformer-decoder model,” Journal of Chemical Information and Modeling, vol. 62, no. 9, pp. 2064–2076, 2021.
  * [78] R. Irwin, S. Dimitriadis, J. He, and E. J. Bjerrum, “Chemformer: a pre-trained transformer for computational chemistry,” Machine Learning: Science and Technology, vol. 3, no. 1, p. 015022, 2022.
  * [79] Z. Zeng, B. Yin, S. Wang, J. Liu, C. Yang, H. Yao, X. Sun, M. Sun, G. Xie, and Z. Liu, “Interactive molecular discovery with natural language,” arXiv preprint arXiv:2306.11976, 2023.

## Appendix

### A. The Details of Models

An overview of model classifications and architectures is shown in Table 2.

Table 2: Overview of models in molecular modeling design categorized by their
architecture and backbone. The abbreviations MC, INR, MPP, MR, MIR, MG, and OT
correspond to the tasks investigated: molecule captioning, IUPAC name
recognition, molecular property prediction, molecular retrieval, molecular
image recognition, molecule generation, and other tasks, respectively. Each
$\bullet$ symbol in the table denotes the demonstrated task capabilities of
the models as reported in their publications. Notably, the  $\bullet$ symbol
specifically signifies the specialized task of molecular optimization within
the broader context of molecule generation.

Architecture | Backbone | Model | Date | MC | INR | MPP | MR | MIR | MG | OT
---|---|---|---|---|---|---|---|---|---|---
Encoder-only | BERT |  KV-PLM [27] | 2022/02 |  |  | $\bullet$ | $\bullet$ |  |  | $\bullet$
CLAMP [48] | 2023/05 |  |  | $\bullet$ |  |  |  |
RoBERTa |  ChemBERTa [38] | 2020/10 |  |  | $\bullet$ |  |  |  |
DMP [49] | 2021/10 |  |  | $\bullet$ |  |  |  |
Transformer |  Molformer [50] | 2022/04 |  |  | $\bullet$ |  |  |  |
MM-Deacon [51] | 2022/04 |  |  | $\bullet$ | $\bullet$ |  |  |
NoiseMol [52] | 2023/01 |  |  | $\bullet$ |  |  |  |
TransFoxMol [53] | 2023/01 |  |  | $\bullet$ |  |  |  |
Graphormer [54] |  CAFE-MPP [55] | 2023/01 |  |  | $\bullet$ |  |  |  |
Decoder-only | GPT |  MolGPT [77] | 2021/10 |  |  |  |  |  | $\bullet$ |
Galactica [66] | 2022/12 |  |  | $\bullet$ |  |  |  | $\bullet$
MolXPT [40] | 2023/05 | $\bullet$ |  | $\bullet$ |  |  | $\bullet$ |
Transformer |  Taiga [56] | 2023/05 |  |  |  |  |  | $\bullet$ |
ChatGPT |  MolReGPT [41] | 2023/06 | $\bullet$ |  |  |  |  | $\bullet$ |
|  LLM4Mol [69] | 2023/07 |  |  | $\bullet$ |  |  |  |
Text-Text | BART [42] |  Chemformer [78] | 2022/02 |  |  |  |  |  | $\bullet$ |
MolGen [57] | 2023/05 |  |  |  |  |  | $\bullet$ |
T5 |  C5T5 [58] | 2021/08 |  |  |  |  |  | $\bullet$ |
T5Chem [59] | 2022/03 |  |  |  |  |  |  | $\bullet$
MolT5 [43] | 2022/11 | $\bullet$ |  |  |  |  | $\bullet$ |
Text+Chem T5[44] | 2023/05 | $\bullet$ |  |  |  |  | $\bullet$ | $\bullet$
ChatMol [79] | 2023/06 | $\bullet$ |  |  |  |  | $\bullet$ |
TransAntivirus [68] | 2023/06 |  | $\bullet$ |  |  |  | $\bullet$ |
BioT5 [76] | 2023/10 | $\bullet$ |  | $\bullet$ |  |  | $\bullet$ |
nach0 [60] | 2023/11 | $\bullet$ |  | $\bullet$ |  |  | $\bullet$ | $\bullet$
Transformer |  Struct2IUPAC [67] | 2020/11 |  | $\bullet$ |  |  |  |  |
Transformer-R [61] | 2021/04 |  |  |  |  |  | $\bullet$ |
ChemCrow[62] | 2023/04 |  |  |  |  |  | $\bullet$ | $\bullet$
Image-Text | Swin Transformer [74] |  SwinOCSR [73] | 2022/07 |  |  |  |  | $\bullet$ |  |
MolScribe [75] | 2023/03 |  |  |  |  | $\bullet$ |  |
Transformer |  DECIMER1.0 [70] | 2021/08 |  |  |  |  | $\bullet$ |  |
Image2SMILES[71] | 2022/01 |  |  |  |  | $\bullet$ |  |
MICER [72] | 2022/08 |  |  |  |  | $\bullet$ |  |
Graph-Text | Sci-BERT |  Text2Mol [28] | 2021/09 |  |  |  | $\bullet$ |  |  |
MoMu [25] | 2022/09 | $\bullet$ |  | $\bullet$ | $\bullet$ |  | $\bullet$ |
MegaMolBART [78] |  MoleculeSTM [29] | 2022/12 |  |  | $\bullet$ | $\bullet$ |  | $\bullet$ |
BERT |  MolFM [64] | 2023/07 | $\bullet$ |  | $\bullet$ | $\bullet$ |  | $\bullet$ |
| Q-Former [45] |  MolCA [65] | 2023/11 | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ |  |  |
| KV-PLM |  MolLM [63] | 2023/12 | $\bullet$ |  | $\bullet$ | $\bullet$ |  | $\bullet$ |
Image-Graph-Text | Q-Former |  GIT-Mol [47] | 2023/08 | $\bullet$ |  | $\bullet$ |  | $\bullet$ | $\bullet$ |

### B. ChEBI-20-MM

In our study, we employ visualization techniques to analyze the suitability of
data sources for language models and chemical space coverage. As shown in
Figure 6, we use different visualization methods to analyze the distribution
of text lengths and the number of tokens generated by each model’s tokenizer
for various text data types. This approach allows us to evaluate the
adaptability of language models to the textual characteristics of our dataset.
We also focus on the top 10 scaffolds within the dataset, counting the number
of molecules for each scaffold. Here, semi-transparent bars represent the
total count, while solid bars indicate the quantity in the training set. On
the other hand, for the analysis of chemical space coverage, we choose
molecular weight (MW), LogP, the number of aromatic rings, and the Topological
Polar Surface Area (TPSA) as descriptors. We examine the distribution and
correlation of these descriptors within the dataset, providing insights into
the chemical diversity and complexity present in our data.

Figure 6: Visualization of Data Source Suitability and Chemical Space
Diversity.

A benchmark must exhibit a balanced distribution, preventing biases toward any
specific molecular representation. Analyzing the distribution of text lengths
and tokenizer outputs helps confirm that a language model’s tokenizer captures
molecular information effectively. Moreover, the diversity in scaffold
distribution is crucial for determining the benchmark’s generalization across
chemical spaces, ensuring the language model can learn and predict a wide
array of chemical entities. Finally, the distribution and correlation of
molecular descriptors reveal the complexity within the chemical space.
Overall, the visual analyses of text length, scaffold count, and descriptor
statistics provide insights into the dataset’s characteristics, affirming the
benchmark’s robustness for tasks in molecular science.

As for the metrics, we use a combination of BLEU-2, BLEU-4, ROUGE-1, ROUGE-2,
ROUGE-L, and METEOR scores for IUPAC name recognition and molecule captioning.
Moreover, for image recognition and molecule generation, we assess models
using BLEU for textual similarity, Exact Match for precision, Valid for
chemical validity of generated molecules, Levenshtein distance for structural
similarity, and fingerprint-based metrics such as MACCS,
RDKit111https://www.rdkit.org/, and Morgan for capturing chemical feature
representation. Molecular Retrieval is evaluated using Mean Reciprocal Rank
(MRR), and Recall at K (R@1, R@5, R@10). For the property prediction metrics,
We utilize widely-recognized classification and regression tasks from
MoleculeNet. For classification, we measure the area under the receiver
operating characteristic curve (roc_auc), the area under the precision-recall
curve (pr_auc), and the mean F1 score (f1_score). For regression tasks, we use
mean squared error (mse_mean), root mean squared error (rmse_mean), and mean
absolute error (mae_mean) to assess the predictive accuracy of the models
quantitatively.

### C. Experimental Results

#### SMILES Generation.

Table 3 illustrates the performance of SMILES generation across two
methodologies: image recognition and text-based SMILES generation. The image
recognition task is an essential task for chemical image understanding. The
results show that models leveraging Swin Transformer (SwinOCSR) for encoding
and t5 for decoding effectively generate SMILES from visual inputs.

Table 3: Experimental Results for SMILES Generation

Task | Input | Encoder | Decoder | Bleu $\uparrow$ | Exact Match $\uparrow$ | Levenshtein $\downarrow$ | MACCS $\uparrow$ | RDK $\uparrow$ | Morgan $\uparrow$ | Validity $\uparrow$
---|---|---|---|---|---|---|---|---|---|---
image recognition (SMILES) | image | swinocsr | t5 | 0.885 | 0.356 | 12.746 | 0.951 | 0.890 | 0.868 | 0.908
image | swinocsr | t511 | 0.833 | 0.175 | 18.589 | 0.897 | 0.785 | 0.765 | 0.761
image | vit | t5 | 0.826 | 0.179 | 19.802 | 0.889 | 0.780 | 0.734 | 0.873
image | swin | t5 | 0.782 | 0.067 | 24.726 | 0.794 | 0.633 | 0.581 | 0.891
image | swin | t511 | 0.779 | 0.066 | 25.203 | 0.802 | 0.648 | 0.597 | 0.850
molecule generation (text-based) | IUPAC | t5 | t5 | 0.881 | 0.327 | 14.209 | 0.909 | 0.811 | 0.773 | 0.905
IUPAC | t511 | t511 | 0.881 | 0.323 | 14.141 | 0.911 | 0.808 | 0.771 | 0.893
IUPAC | molt5 | molt5 | 0.853 | 0.206 | 17.645 | 0.881 | 0.758 | 0.723 | 0.773
IUPAC | scibert | molt5 | 0.614 | 0.054 | 51.003 | 0.772 | 0.590 | 0.526 | 0.680
IUPAC | bert | t5 | 0.296 | 0.000 | 96.484 | 0.409 | 0.268 | 0.128 | 0.988
caption | molt5 | molt5 | 0.754 | 0.090 | 28.034 | 0.808 | 0.673 | 0.594 | 0.837
caption | t5 | t5 | 0.691 | 0.097 | 34.891 | 0.814 | 0.677 | 0.600 | 0.858
caption | t511 | t511 | 0.649 | 0.110 | 40.569 | 0.819 | 0.687 | 0.608 | 0.857
caption | scibert | molt5 | 0.543 | 0.012 | 57.118 | 0.670 | 0.492 | 0.431 | 0.693
caption | chemberta | molt5 | 0.425 | 0.002 | 73.061 | 0.561 | 0.408 | 0.325 | 0.740

Within the text-based SMILES generation, the combination of IUPAC nomenclature
with the t5 model emerges as the most effective, highlighting the model’s
ability to process and convert complex chemical language into accurate SMILES
structures. Moreover, the molt5 model’s superior performance in generating
molecules from captions suggests that its pre-training on SMILES expressions
is advantageous for this specific task, emphasizing the value of domain-
specific pre-training in enhancing model performance.

Table 4: Experimental Results for IUPAC and Caption Generation

Task | Input | Encoder | Decoder | Bleu-2 $\uparrow$ | Bleu-4 $\uparrow$ | Rouge-1 $\uparrow$ | Rouge-2 $\uparrow$ | Rouge-L $\uparrow$ | Meteor-score $\uparrow$ | Exact Match $\uparrow$
---|---|---|---|---|---|---|---|---|---|---
IUPAC recognition (image) | image | swinocsr | t5 | 0.820 | 0.758 | 0.781 | 0.621 | 0.740 | 0.799 | 0.220
image | swinocsr | molt5 | 0.803 | 0.730 | 0.751 | 0.573 | 0.707 | 0.777 | 0.158
image | swinocsr | t511 | 0.800 | 0.730 | 0.757 | 0.577 | 0.711 | 0.779 | 0.170
image | vit | t511 | 0.745 | 0.655 | 0.657 | 0.445 | 0.603 | 0.677 | 0.088
image | vit | molt5 | 0.725 | 0.632 | 0.644 | 0.431 | 0.589 | 0.667 | 0.069
IUPAC recognition | SMILES | t5 | t5 | 0.820 | 0.755 | 0.751 | 0.573 | 0.703 | 0.767 | 0.210
SMILES | t511 | t511 | 0.794 | 0.719 | 0.711 | 0.515 | 0.660 | 0.732 | 0.146
SMILES | chemberta | molt5 | 0.786 | 0.710 | 0.717 | 0.524 | 0.669 | 0.739 | 0.134
SMILES | molt5 | molt5 | 0.752 | 0.665 | 0.669 | 0.459 | 0.615 | 0.688 | 0.095
SMILES | bart | bart | 0.598 | 0.500 | 0.684 | 0.463 | 0.624 | 0.676 | 0.170
InChI | t5 | t5 | 0.744 | 0.658 | 0.648 | 0.442 | 0.591 | 0.663 | 0.131
InChI | t511 | t511 | 0.728 | 0.636 | 0.622 | 0.410 | 0.565 | 0.641 | 0.088
InChI | molt5 | molt5 | 0.728 | 0.635 | 0.622 | 0.413 | 0.566 | 0.642 | 0.093
InChI | chemberta | molt5 | 0.646 | 0.541 | 0.550 | 0.326 | 0.493 | 0.574 | 0.037
InChI | roberta | molt5 | 0.620 | 0.505 | 0.494 | 0.249 | 0.429 | 0.509 | 0.012
SELFIES | molt5 | molt5 | 0.751 | 0.666 | 0.665 | 0.458 | 0.611 | 0.687 | 0.111
SELFIES | t5 | t5 | 0.744 | 0.658 | 0.668 | 0.465 | 0.615 | 0.692 | 0.132
SELFIES | t511 | t511 | 0.720 | 0.633 | 0.655 | 0.440 | 0.601 | 0.677 | 0.097
SELFIES | biogpt | biogpt | 0.669 | 0.585 | 0.594 | 0.391 | 0.534 | 0.602 | 0.005
SELFIES | bart | bart | 0.550 | 0.433 | 0.568 | 0.311 | 0.501 | 0.569 | 0.063
graph | gcn | t511 | 0.719 | 0.643 | 0.640 | 0.436 | 0.591 | 0.667 | 0.056
graph | gat | t5 | 0.708 | 0.630 | 0.620 | 0.420 | 0.572 | 0.652 | 0.046
graph | gat | t511 | 0.685 | 0.603 | 0.597 | 0.393 | 0.550 | 0.628 | 0.037
graph | gin | t5 | 0.682 | 0.600 | 0.615 | 0.403 | 0.558 | 0.653 | 0.052
graph | gin | molt5 | 0.662 | 0.580 | 0.605 | 0.395 | 0.555 | 0.643 | 0.029
| caption | t511 | t511 | 0.736 | 0.665 | 0.707 | 0.532 | 0.660 | 0.716 | 0.185
| caption | biogpt | biogpt | 0.718 | 0.655 | 0.650 | 0.479 | 0.605 | 0.673 | 0.000
| caption | molt5 | molt5 | 0.699 | 0.615 | 0.652 | 0.460 | 0.604 | 0.667 | 0.129
| caption | t5 | t5 | 0.621 | 0.545 | 0.675 | 0.495 | 0.627 | 0.679 | 0.183
| caption | scibert | molt5 | 0.596 | 0.503 | 0.531 | 0.314 | 0.478 | 0.568 | 0.025
molecule captioning | IUPAC | t5 | t5 | 0.547 | 0.457 | 0.629 | 0.476 | 0.566 | 0.579 | /
IUPAC | t511 | t511 | 0.535 | 0.446 | 0.622 | 0.469 | 0.559 | 0.571 | /
IUPAC | molt5 | molt5 | 0.525 | 0.433 | 0.612 | 0.455 | 0.549 | 0.556 | /
IUPAC | bart | bart | 0.455 | 0.352 | 0.568 | 0.391 | 0.489 | 0.484 | /
IUPAC | chemberta | molt5 | 0.424 | 0.333 | 0.550 | 0.386 | 0.492 | 0.490 | /
SMILES | molt5 | molt5 | 0.535 | 0.436 | 0.625 | 0.466 | 0.559 | 0.563 | /
SMILES | t5 | t5 | 0.508 | 0.414 | 0.605 | 0.446 | 0.545 | 0.543 | /
SMILES | t511 | t511 | 0.488 | 0.392 | 0.586 | 0.426 | 0.528 | 0.522 | /
SMILES | chemberta | molt5 | 0.488 | 0.388 | 0.578 | 0.417 | 0.522 | 0.520 | /
SMILES | bart | bart | 0.406 | 0.301 | 0.538 | 0.359 | 0.465 | 0.443 | /
InChI | t5 | t5 | 0.477 | 0.383 | 0.577 | 0.415 | 0.518 | 0.511 | /
InChI | t511 | t511 | 0.464 | 0.369 | 0.567 | 0.404 | 0.510 | 0.499 | /
InChI | molt5 | molt5 | 0.461 | 0.367 | 0.571 | 0.406 | 0.513 | 0.500 | /
InChI | chemberta | molt5 | 0.451 | 0.356 | 0.557 | 0.394 | 0.502 | 0.490 | /
InChI | bart | bart | 0.393 | 0.284 | 0.519 | 0.335 | 0.447 | 0.429 | /
SELFIES | t5 | t5 | 0.462 | 0.360 | 0.569 | 0.400 | 0.506 | 0.503 | /
SELFIES | t511 | t511 | 0.446 | 0.348 | 0.559 | 0.392 | 0.499 | 0.490 | /
SELFIES | chemberta | molt5 | 0.439 | 0.340 | 0.549 | 0.381 | 0.491 | 0.478 | /
SELFIES | bart | bart | 0.377 | 0.266 | 0.510 | 0.320 | 0.434 | 0.414 | /
SELFIES | biogpt | biogpt | 0.277 | 0.204 | 0.425 | 0.250 | 0.355 | 0.345 | /
graph | gat | molt5 | 0.419 | 0.318 | 0.521 | 0.345 | 0.460 | 0.449 | /
graph | gin | molt5 | 0.407 | 0.305 | 0.512 | 0.332 | 0.450 | 0.437 | /
graph | gcn | molt5 | 0.392 | 0.291 | 0.512 | 0.333 | 0.450 | 0.430 | /
graph | gin | t511 | 0.362 | 0.256 | 0.479 | 0.295 | 0.418 | 0.401 | /
graph | gcn | bart | 0.358 | 0.251 | 0.489 | 0.298 | 0.413 | 0.391 | /
| image | swinocsr | molt5 | 0.495 | 0.399 | 0.590 | 0.429 | 0.532 | 0.527 | /
| image | swinocsr | t511 | 0.492 | 0.394 | 0.585 | 0.422 | 0.527 | 0.524 | /
| image | swinocsr | t5 | 0.417 | 0.317 | 0.533 | 0.361 | 0.472 | 0.461 | /
| image | swin | molt5 | 0.409 | 0.307 | 0.517 | 0.343 | 0.460 | 0.442 | /
| image | swin | t511 | 0.400 | 0.301 | 0.517 | 0.344 | 0.461 | 0.435 | /

#### IUPAC Generation.

Table 4 emphasizes the tasks associated with IUPAC Generation, focusing on the
recognition of IUPAC names from molecular images and internal information. Our
benchmark does not include experiments for generating IUPAC names from
captions because some captions already contain IUPAC names.

Table 5: Experimental Results for Retrieval

Task | Input | Encoder | Pooling | Target Modality | R@1 $\uparrow$ | MRR $\uparrow$ | R@5 $\uparrow$ | R@10 $\uparrow$
---|---|---|---|---|---|---|---|---
molecular retrieval (caption) | IUPAC | t5 | max | caption | ${\color[rgb]{1,0,0}\textbf{0.637}}\pm 0.024$ | ${\color[rgb]{1,0,0}\textbf{0.741}}\pm 0.022$ | ${\color[rgb]{1,0,0}\textbf{0.880}}\pm 0.019$ | ${\color[rgb]{1,0,0}\textbf{0.936}}\pm 0.012$
IUPAC | scibert | avg | caption | $0.635\pm 0.084$ | $0.736\pm 0.076$ | $0.870\pm 0.063$ | $0.935\pm 0.051$
IUPAC | scibert | max | caption | $0.581\pm 0.061$ | $0.694\pm 0.056$ | $0.848\pm 0.046$ | $0.918\pm 0.036$
IUPAC | molt5 | avg | caption | $0.578\pm 0.026$ | $0.701\pm 0.026$ | $0.869\pm 0.027$ | $0.931\pm 0.016$
IUPAC | t5 | avg | caption | $0.564\pm 0.035$ | $0.685\pm 0.034$ | $0.855\pm 0.032$ | $0.923\pm 0.022$
SMILES | bart | avg | caption | $\textbf{0.440}\pm 0.043$ | $\textbf{0.589}\pm 0.043$ | $0.797\pm 0.044$ | $0.895\pm 0.033$
SMILES | t5 | max | caption | $0.437\pm 0.046$ | $0.588\pm 0.046$ | $\textbf{0.800}\pm 0.045$ | $0.892\pm 0.028$
SMILES | t5 | avg | caption | $0.427\pm 0.056$ | $0.583\pm 0.056$ | $0.797\pm 0.056$ | $\textbf{0.897}\pm 0.037$
SMILES | roberta | max | caption | $0.426\pm 0.070$ | $0.572\pm 0.075$ | $0.772\pm 0.082$ | $0.880\pm 0.066$
SMILES | molt5 | avg | caption | $0.402\pm 0.054$ | $0.557\pm 0.055$ | $0.772\pm 0.059$ | $0.875\pm 0.037$
InChI | t5 | avg | caption | $\textbf{0.339}\pm 0.033$ | $\textbf{0.504}\pm 0.036$ | $\textbf{0.733}\pm 0.040$ | $\textbf{0.858}\pm 0.030$
InChI | scibert | avg | caption | $0.337\pm 0.076$ | $0.498\pm 0.095$ | $0.719\pm 0.121$ | $0.843\pm 0.111$
InChI | t511 | avg | caption | $0.325\pm 0.053$ | $0.488\pm 0.062$ | $0.717\pm 0.077$ | $0.846\pm 0.062$
InChI | bart | avg | caption | $0.289\pm 0.036$ | $0.444\pm 0.045$ | $0.661\pm 0.059$ | $0.808\pm 0.051$
InChI | molt5 | avg | caption | $0.280\pm 0.055$ | $0.433\pm 0.064$ | $0.646\pm 0.079$ | $0.790\pm 0.068$
SELFIES | roberta | max | caption | $\textbf{0.436}\pm 0.079$ | $\textbf{0.579}\pm 0.081$ | $\textbf{0.778}\pm 0.086$ | $\textbf{0.875}\pm 0.069$
SELFIES | scibert | max | caption | $0.406\pm 0.053$ | $0.553\pm 0.057$ | $0.757\pm 0.063$ | $\textbf{0.875}\pm 0.051$
SELFIES | bert | max | caption | $0.393\pm 0.055$ | $0.546\pm 0.063$ | $0.751\pm 0.073$ | $0.868\pm 0.062$
SELFIES | bart | avg | caption | $0.367\pm 0.053$ | $0.520\pm 0.057$ | $0.731\pm 0.065$ | $0.853\pm 0.055$
SELFIES | molt5 | max | caption | $0.339\pm 0.041$ | $0.492\pm 0.045$ | $0.709\pm 0.054$ | $0.835\pm 0.045$
graph | gat | max | caption | $\textbf{0.259}\pm 0.060$ | $\textbf{0.406}\pm 0.079$ | $\textbf{0.610}\pm 0.111$ | $\textbf{0.765}\pm 0.111$
graph | gcn | max | caption | $0.242\pm 0.048$ | $0.383\pm 0.059$ | $0.576\pm 0.077$ | $0.724\pm 0.068$
graph | gin | max | caption | $0.241\pm 0.030$ | $0.384\pm 0.040$ | $0.577\pm 0.053$ | $0.732\pm 0.053$
graph | gat | avg | caption | $0.189\pm 0.046$ | $0.319\pm 0.068$ | $0.501\pm 0.102$ | $0.648\pm 0.108$
graph | gin | avg | caption | $0.185\pm 0.038$ | $0.309\pm 0.058$ | $0.476\pm 0.086$ | $0.630\pm 0.099$
| image | swinocsr | max | caption | $\textbf{0.411}\pm 0.124$ | $\textbf{0.552}\pm 0.146$ | $\textbf{0.748}\pm 0.181$ | $0.844\pm 0.164$
| image | swinocsr | avg | caption | $0.384\pm 0.094$ | $0.534\pm 0.106$ | $0.747\pm 0.127$ | $\textbf{0.856}\pm 0.107$
| image | resnet | avg | caption | $0.233\pm 0.041$ | $0.385\pm 0.055$ | $0.587\pm 0.075$ | $0.753\pm 0.073$
| image | vit | max | caption | $0.164\pm 0.045$ | $0.285\pm 0.068$ | $0.443\pm 0.099$ | $0.607\pm 0.116$
| image | vit | avg | caption | $0.121\pm 0.019$ | $0.239\pm 0.034$ | $0.399\pm 0.056$ | $0.574\pm 0.070$
molecular retrieval (IUPAC) | SMILES | t5 | avg | IUPAC | ${\color[rgb]{1,0,0}\textbf{0.709}}\pm 0.069$ | ${\color[rgb]{1,0,0}\textbf{0.817}}\pm 0.054$ | ${\color[rgb]{1,0,0}\textbf{0.958}}\pm 0.033$ | ${\color[rgb]{1,0,0}\textbf{0.986}}\pm 0.015$
SMILES | t511 | max | IUPAC | $0.698\pm 0.116$ | $0.802\pm 0.096$ | $0.941\pm 0.065$ | $0.975\pm 0.032$
SMILES | scibert | avg | IUPAC | $0.681\pm 0.147$ | $0.789\pm 0.127$ | $0.930\pm 0.097$ | $0.967\pm 0.060$
SMILES | t511 | avg | IUPAC | $0.655\pm 0.099$ | $0.775\pm 0.082$ | $0.930\pm 0.055$ | $0.970\pm 0.028$
SMILES | bart | max | IUPAC | $0.629\pm 0.090$ | $0.759\pm 0.075$ | $0.935\pm 0.051$ | $0.978\pm 0.025$
InChI | t5 | avg | IUPAC | $\textbf{0.545}\pm 0.064$ | $\textbf{0.687}\pm 0.059$ | $0.877\pm 0.054$ | $0.945\pm 0.028$
InChI | t5 | max | IUPAC | $0.519\pm 0.074$ | $0.661\pm 0.065$ | $0.850\pm 0.051$ | $0.931\pm 0.031$
InChI | t511 | avg | IUPAC | $0.518\pm 0.114$ | $0.665\pm 0.111$ | $0.867\pm 0.106$ | $0.936\pm 0.073$
InChI | bart | avg | IUPAC | $0.502\pm 0.085$ | $0.662\pm 0.084$ | $\textbf{0.882}\pm 0.085$ | $\textbf{0.950}\pm 0.049$
InChI | t511 | max | IUPAC | $0.499\pm 0.099$ | $0.654\pm 0.094$ | $0.868\pm 0.090$ | $0.945\pm 0.046$
SELFIES | t511 | max | IUPAC | $\textbf{0.626}\pm 0.068$ | $\textbf{0.753}\pm 0.057$ | $\textbf{0.920}\pm 0.038$ | $\textbf{0.972}\pm 0.021$
SELFIES | bart | avg | IUPAC | $0.603\pm 0.094$ | $0.731\pm 0.083$ | $0.906\pm 0.067$ | $0.962\pm 0.038$
SELFIES | t5 | avg | IUPAC | $0.570\pm 0.076$ | $0.706\pm 0.069$ | $0.887\pm 0.059$ | $0.950\pm 0.035$
SELFIES | bart | max | IUPAC | $0.508\pm 0.095$ | $0.662\pm 0.094$ | $0.876\pm 0.097$ | $0.947\pm 0.063$
SELFIES | molt5 | avg | IUPAC | $0.506\pm 0.112$ | $0.643\pm 0.112$ | $0.828\pm 0.114$ | $0.912\pm 0.080$
graph | gcn | max | IUPAC | $\textbf{0.279}\pm 0.067$ | $0.421\pm 0.084$ | $0.621\pm 0.113$ | $0.756\pm 0.107$
graph | gin | max | IUPAC | $0.272\pm 0.068$ | $\textbf{0.423}\pm 0.088$ | $\textbf{0.631}\pm 0.118$ | $\textbf{0.773}\pm 0.114$
graph | gin | avg | IUPAC | $0.216\pm 0.071$ | $0.352\pm 0.104$ | $0.536\pm 0.152$ | $0.684\pm 0.169$
graph | gat | avg | IUPAC | $0.204\pm 0.074$ | $0.336\pm 0.112$ | $0.518\pm 0.170$ | $0.669\pm 0.191$
graph | gcn | avg | IUPAC | $0.186\pm 0.030$ | $0.321\pm 0.046$ | $0.500\pm 0.068$ | $0.661\pm 0.080$
image | swin | max | IUPAC | $\textbf{0.696}\pm 0.083$ | $\textbf{0.803}\pm 0.069$ | $\textbf{0.945}\pm 0.050$ | $\textbf{0.979}\pm 0.030$
image | swin | avg | IUPAC | $0.462\pm 0.104$ | $0.608\pm 0.105$ | $0.808\pm 0.104$ | $0.900\pm 0.076$
image | resnet | avg | IUPAC | $0.289\pm 0.057$ | $0.450\pm 0.070$ | $0.674\pm 0.091$ | $0.834\pm 0.080$
image | resnet | max | IUPAC | $0.215\pm 0.070$ | $0.376\pm 0.110$ | $0.598\pm 0.170$ | $0.761\pm 0.188$
image | vit | max | IUPAC | $0.215\pm 0.080$ | $0.365\pm 0.117$ | $0.575\pm 0.172$ | $0.740\pm 0.178$

For IUPAC name recognition from the molecular images, the combination of
swinocsr and t5 models yields the best performance, similar to the SMILES
generation task. It suggests that the swinocsr, originally pre-trained on
molecular SMILES and images, is also adept at transfer learning for IUPAC name
recognition tasks. On the other hand, combining SMILES data with the t5 model
achieves the best results for IUPAC recognition based on molecular internal
information. Although graph-based models show only slightly lower performance
in textual similarity metrics compared to language models, they exhibit a
relative shortfall in achieving exact matches.

#### Molecule Captioning.

The molecular captions encapsulate the essence of the molecule along with its
biochemical and physicochemical properties. Molecule captioning involves the
conversion of various representation forms into captions. As shown in Table 4,
unlike image recognition tasks for SMILES and IUPAC names, experiments on
caption generation show that performance based on IUPAC and SMILES outperforms
those based on image-derived captions. It suggests that image encoders excel
at distilling key points from images. However, as the diversity of generated
content increases and the length of the generated sequences extends, the
performance of image encoder models tends to fall short of language models.
Within the scope of language models, text description generation based on
IUPAC is superior, partly because the target captions commonly include IUPAC
names. Additionally, graph encoders exhibit weaker performance in text
generation tasks, further emphasizing the strengths of language models in
handling complex linguistic structures and content variability.

Table 6: Experimental Results for Classification Tasks

Task | Data Spliting | Dataset | Input | Encoder | Pooling | roc_auc $\uparrow$ | pr_auc $\uparrow$ | f1_score $\uparrow$
---|---|---|---|---|---|---|---|---
property prediction (classification) | scaffold | bace | SELFIES | roberta | max | $\textbf{0.742}\pm 0.036$ | $\textbf{0.750}\pm 0.013$ | $0.544\pm 0.126$
bace | SELFIES | molt5 | max | $0.708\pm 0.065$ | $0.678\pm 0.043$ | $0.391\pm 0.033$
bace | InChI | bart | avg | $0.704\pm 0.058$ | $0.699\pm 0.063$ | $0.516\pm 0.112$
bace | InChI | t5 | max | $0.703\pm 0.030$ | $0.699\pm 0.043$ | $\textbf{0.646}\pm 0.032$
bace | InChI | molt5 | max | $0.678\pm 0.039$ | $0.699\pm 0.036$ | $0.565\pm 0.038$
bbbp | graph | gin | avg | $\textbf{0.652}\pm 0.009$ | $0.684\pm 0.024$ | $0.528\pm 0.026$
bbbp | graph | gat | avg | $0.651\pm 0.013$ | $0.691\pm 0.013$ | $0.470\pm 0.057$
bbbp | SMILES | molt5 | avg | $0.651\pm 0.010$ | $0.692\pm 0.010$ | $\textbf{0.602}\pm 0.034$
bbbp | graph | gcn | max | $0.645\pm 0.011$ | $\textbf{0.693}\pm 0.013$ | $0.492\pm 0.118$
bbbp | graph | gcn | avg | $0.641\pm 0.012$ | $0.675\pm 0.014$ | $0.536\pm 0.057$
clintox | SELFIES | t511 | avg | $\textbf{0.923}\pm 0.003$ | $0.722\pm 0.047$ | $\textbf{0.483}\pm 0.000$
clintox | InChI | t5 | avg | $0.908\pm 0.011$ | $0.700\pm 0.022$ | $0.483\pm 0.000$
clintox | SELFIES | t5 | avg | $0.899\pm 0.064$ | $\textbf{0.749}\pm 0.096$ | $\textbf{0.483}\pm 0.000$
clintox | SELFIES | t5 | max | $0.874\pm 0.020$ | $0.721\pm 0.034$ | $0.381\pm 0.144$
clintox | InChI | molt5 | avg | $0.874\pm 0.039$ | $0.667\pm 0.039$ | $\textbf{0.483}\pm 0.000$
sider | SMILES | t511 | avg | $\textbf{0.563}\pm 0.029$ | $0.606\pm 0.006$ | $0.426\pm 0.026$
sider | graph | gat | avg | $0.557\pm 0.007$ | $0.605\pm 0.003$ | $0.442\pm 0.014$
sider | graph | gat | max | $0.551\pm 0.021$ | $0.598\pm 0.014$ | $0.455\pm 0.010$
sider | InChI | t5 | avg | $0.550\pm 0.016$ | $0.601\pm 0.014$ | $\textbf{0.459}\pm 0.011$
sider | graph | gcn | max | $0.545\pm 0.026$ | $\textbf{0.607}\pm 0.016$ | $0.443\pm 0.003$
tox21 | graph | gin | avg | $\textbf{0.693}\pm 0.019$ | $\textbf{0.248}\pm 0.037$ | $0.473\pm 0.000$
tox21 | graph | gcn | avg | $0.679\pm 0.007$ | $0.231\pm 0.007$ | $0.473\pm 0.000$
tox21 | SMILES | molt5 | avg | $0.664\pm 0.031$ | $0.200\pm 0.058$ | $\textbf{0.488}\pm 0.021$
tox21 | graph | gat | avg | $0.664\pm 0.014$ | $0.212\pm 0.003$ | $0.473\pm 0.000$
tox21 | SMILES | t511 | avg | $0.657\pm 0.043$ | $0.212\pm 0.050$ | $0.486\pm 0.018$
toxcast | graph | gat | avg | $\textbf{0.606}\pm 0.005$ | $0.305\pm 0.002$ | $0.449\pm 0.003$
toxcast | graph | gcn | avg | $0.604\pm 0.006$ | $\textbf{0.306}\pm 0.002$ | $0.452\pm 0.003$
toxcast | graph | gin | avg | $0.599\pm 0.008$ | $0.294\pm 0.016$ | $0.446\pm 0.001$
toxcast | SMILES | t511 | avg | $0.582\pm 0.008$ | $0.302\pm 0.006$ | $0.451\pm 0.007$
toxcast | SELFIES | t511 | avg | $0.579\pm 0.002$ | $0.293\pm 0.002$ | $\textbf{0.453}\pm 0.002$
random | bace | graph | gcn | avg | $\textbf{0.824}\pm 0.034$ | $\textbf{0.752}\pm 0.023$ | $\textbf{0.683}\pm 0.142$
bace | graph | gat | avg | $0.782\pm 0.022$ | $0.715\pm 0.037$ | $0.398\pm 0.039$
bace | graph | gin | avg | $0.751\pm 0.034$ | $0.684\pm 0.061$ | $0.399\pm 0.042$
bace | SMILES | t5 | avg | $0.712\pm 0.043$ | $0.609\pm 0.056$ | $0.429\pm 0.089$
bace | InChI | t511 | avg | $0.710\pm 0.036$ | $0.656\pm 0.035$ | $0.459\pm 0.065$
bbbp | graph | gcn | avg | $\textbf{0.872}\pm 0.014$ | $0.935\pm 0.017$ | $\textbf{0.769}\pm 0.031$
bbbp | SMILES | molt5 | avg | $0.849\pm 0.062$ | $0.928\pm 0.036$ | $0.757\pm 0.071$
bbbp | graph | gat | avg | $0.848\pm 0.020$ | $0.932\pm 0.013$ | $0.768\pm 0.012$
bbbp | SMILES | t511 | avg | $0.848\pm 0.015$ | $\textbf{0.940}\pm 0.002$ | $0.742\pm 0.016$
bbbp | SMILES | t5 | avg | $0.847\pm 0.027$ | $0.932\pm 0.017$ | $0.756\pm 0.003$
clintox | SMILES | t5 | avg | $\textbf{0.861}\pm 0.041$ | $\textbf{0.654}\pm 0.029$ | $\textbf{0.485}\pm 0.000$
clintox | SMILES | molt5 | avg | $0.852\pm 0.023$ | $0.643\pm 0.026$ | $0.478\pm 0.004$
clintox | SELFIES | t5 | avg | $0.850\pm 0.004$ | $0.611\pm 0.027$ | $0.484\pm 0.003$
clintox | SMILES | t511 | avg | $0.845\pm 0.063$ | $0.652\pm 0.088$ | $0.482\pm 0.009$
clintox | SELFIES | t511 | avg | $0.839\pm 0.022$ | $0.607\pm 0.009$ | $0.483\pm 0.005$
sider | SMILES | t5 | avg | $\textbf{0.581}\pm 0.031$ | $\textbf{0.629}\pm 0.015$ | $\textbf{0.475}\pm 0.017$
sider | graph | gat | max | $0.580\pm 0.018$ | $0.600\pm 0.002$ | $0.439\pm 0.013$
sider | InChI | t5 | avg | $0.568\pm 0.038$ | $0.618\pm 0.017$ | $0.461\pm 0.015$
sider | SMILES | molt5 | avg | $0.562\pm 0.008$ | $0.600\pm 0.008$ | $0.461\pm 0.012$
sider | graph | gin | max | $0.561\pm 0.018$ | $0.618\pm 0.011$ | $0.451\pm 0.008$
tox21 | graph | gcn | avg | $\textbf{0.795}\pm 0.023$ | $\textbf{0.283}\pm 0.021$ | $0.481\pm 0.001$
tox21 | graph | gin | avg | $0.752\pm 0.025$ | $0.224\pm 0.025$ | $0.480\pm 0.001$
tox21 | SMILES | t511 | avg | $0.745\pm 0.008$ | $0.246\pm 0.041$ | $\textbf{0.494}\pm 0.021$
tox21 | SMILES | molt5 | avg | $0.738\pm 0.011$ | $0.268\pm 0.011$ | $0.492\pm 0.013$
tox21 | graph | gin | max | $0.733\pm 0.016$ | $0.214\pm 0.010$ | $0.480\pm 0.001$
toxcast | SMILES | molt5 | avg | $\textbf{0.673}\pm 0.003$ | $\textbf{0.336}\pm 0.017$ | $0.465\pm 0.007$
toxcast | InChI | t5 | avg | $\textbf{0.673}\pm 0.018$ | $0.322\pm 0.027$ | $\textbf{0.471}\pm 0.001$
toxcast | SELFIES | molt5 | avg | $0.663\pm 0.014$ | $0.319\pm 0.014$ | $0.469\pm 0.004$
toxcast | graph | gcn | avg | $0.662\pm 0.033$ | $0.325\pm 0.030$ | $0.464\pm 0.007$
toxcast | SMILES | t511 | avg | $0.657\pm 0.038$ | $0.318\pm 0.029$ | $0.464\pm 0.009$

#### Molecular Retrieval.

In molecular cross-modal retrieval tasks, the focus is primarily on the
retrieval between molecular external information and either internal or
external information. We categorize the retrieval tasks into two main types,
targeting captions and IUPAC, further subdivided into nine specific retrieval
tasks. As shown in Table 5, the IUPAC-to-caption and SMILES-to-IUPAC tasks
achieve the best results in their respective categories. Regarding the models,
the t5 family demonstrates superiority over bart, encoder-only models, and
graph encoders.

Table 7: Experimental Results for Regression Tasks

Task | Data Spliting | Dataset | Input | Encoder | Pooling | mse $\downarrow$ | rmse $\downarrow$ | mae $\downarrow$
---|---|---|---|---|---|---|---|---
property prediction (regression) | scaffold | esol | SMILES | t5 | avg | $\textbf{1.573}\pm 0.139$ | $1.253\pm 0.056$ | $\textbf{0.972}\pm 0.027$
esol | SMILES | molt5 | avg | $1.593\pm 0.408$ | $\textbf{1.252}\pm 0.158$ | $0.974\pm 0.102$
esol | InChI | molt5 | avg | $1.739\pm 0.089$ | $1.318\pm 0.034$ | $0.965\pm 0.038$
esol | graph | gat | avg | $1.750\pm 0.173$ | $1.321\pm 0.066$ | $0.983\pm 0.057$
esol | SMILES | t511 | avg | $1.910\pm 0.533$ | $1.369\pm 0.192$ | $1.039\pm 0.125$
freesolv | graph | gcn | max | $\textbf{9.701}\pm 1.892$ | $\textbf{3.101}\pm 0.295$ | $\textbf{2.254}\pm 0.225$
freesolv | SELFIES | molt5 | avg | $9.858\pm 0.294$ | $3.139\pm 0.047$ | $2.391\pm 0.007$
freesolv | graph | gat | avg | $10.429\pm 2.184$ | $3.212\pm 0.332$ | $2.608\pm 0.267$
freesolv | SELFIES | t5 | avg | $11.689\pm 0.942$ | $3.416\pm 0.140$ | $2.451\pm 0.113$
freesolv | SELFIES | t511 | avg | $11.740\pm 3.591$ | $3.389\pm 0.504$ | $2.637\pm 0.411$
lipophilicity | graph | gin | avg | $\textbf{0.676}\pm 0.067$ | $\textbf{0.821}\pm 0.040$ | $\textbf{0.656}\pm 0.043$
lipophilicity | graph | gcn | avg | $0.735\pm 0.006$ | $0.857\pm 0.003$ | $0.684\pm 0.007$
lipophilicity | graph | gcn | max | $0.770\pm 0.093$ | $0.876\pm 0.052$ | $0.699\pm 0.046$
lipophilicity | graph | gat | avg | $0.855\pm 0.067$ | $0.924\pm 0.036$ | $0.736\pm 0.025$
lipophilicity | SMILES | molt5 | max | $0.890\pm 0.147$ | $0.940\pm 0.077$ | $0.762\pm 0.064$
random | esol | InChI | t511 | avg | $\textbf{0.967}\pm 0.071$ | $\textbf{0.983}\pm 0.036$ | $\textbf{0.718}\pm 0.053$
esol | SMILES | molt5 | avg | $1.035\pm 0.292$ | $1.007\pm 0.142$ | $0.779\pm 0.130$
esol | SMILES | t511 | avg | $1.201\pm 0.306$ | $1.087\pm 0.138$ | $0.831\pm 0.126$
esol | InChI | molt5 | avg | $1.234\pm 0.157$ | $1.109\pm 0.070$ | $0.794\pm 0.037$
esol | graph | gcn | max | $1.273\pm 0.316$ | $1.120\pm 0.136$ | $0.841\pm 0.103$
freesolv | graph | gcn | max | $\textbf{5.049}\pm 0.493$ | $\textbf{2.244}\pm 0.112$ | $\textbf{1.733}\pm 0.031$
freesolv | graph | gat | max | $5.175\pm 1.119$ | $2.261\pm 0.251$ | $1.818\pm 0.226$
freesolv | graph | gat | avg | $5.536\pm 0.275$ | $2.352\pm 0.058$ | $1.706\pm 0.059$
freesolv | graph | gcn | avg | $6.064\pm 1.485$ | $2.443\pm 0.306$ | $1.829\pm 0.218$
freesolv | InChI | t5 | avg | $7.343\pm 1.255$ | $2.700\pm 0.232$ | $1.920\pm 0.179$
lipophilicity | graph | gin | avg | $\textbf{0.690}\pm 0.099$ | $\textbf{0.828}\pm 0.061$ | $\textbf{0.650}\pm 0.043$
lipophilicity | graph | gat | avg | $0.717\pm 0.019$ | $0.846\pm 0.011$ | $0.676\pm 0.008$
lipophilicity | graph | gcn | avg | $0.785\pm 0.077$ | $0.885\pm 0.043$ | $0.693\pm 0.045$
lipophilicity | graph | gcn | max | $0.795\pm 0.132$ | $0.888\pm 0.075$ | $0.704\pm 0.080$
lipophilicity | SMILES | molt5 | avg | $0.887\pm 0.221$ | $0.934\pm 0.121$ | $0.740\pm 0.100$

#### Property Prediction.

We categorize the tasks into classification and regression, testing them
across scaffold and random data splitting methodologies. The evaluation
extends to nine distinct datasets, each encompassing three text modalities,
eight text encoders, and three graph models, integrated with two different
pooling strategies. Consequently, each dataset is tested with 108 unique
experimental setups. For each combination of input data and model, we select
the top five performing models, as shown in Tables 6 and 7.

The scaffold data splitting method inherently increases the complexity of
predictions. Unlike molecular retrieval tasks, where the focus is on embedding
tasks with textual targets, leading to a better performance of language models
over graph encoders. Here, graph encoders generally outperform language models
in most cases.