# From Artificially Real to Real: Leveraging Pseudo Data from Large Language
Models for Low-Resource Molecule Discovery

Yuhan Chen, Nuwa Xi, Yanrui Du, Haochun Wang, Chen Jianyu, Sendong Zhao, Bing
Qin

###### Abstract

Molecule discovery serves as a cornerstone in numerous scientific domains,
fueling the development of new materials and innovative drug designs. Recent
developments of in-silico molecule discovery have highlighted the promising
results of cross-modal techniques, which bridge molecular structures with
their descriptive annotations. However, these cross-modal methods frequently
encounter the issue of data scarcity, hampering their performance and
application. In this paper, we address the low-resource challenge by utilizing
artificially-real data generated by Large Language Models (LLMs). We first
introduce a retrieval-based prompting strategy to construct high-quality
pseudo data, then explore the optimal method to effectively leverage this
pseudo data. Experiments show that using pseudo data for domain adaptation
outperforms all existing methods, while also requiring a smaller model scale,
reduced data size and lower training cost, highlighting its efficiency.
Furthermore, our method shows a sustained improvement as the volume of pseudo
data increases, revealing the great potential of pseudo data in advancing low-
resource cross-modal molecule discovery.

##  1 Introduction

Molecule discovery plays a critical role in numerous scientific domains
including chemistry (Wang et al. 2023b; Cuzzucoli Crucitti et al. 2023),
pharmacology (Patani and LaVoie 1996; Anderson 2003), and materials science
(Curtarolo et al. 2013). However, traditional molecule design methods are
frequently faced with challenges such as high costs, lengthy development
processes, and limited success rates. Introducing a new drug to the market,
for instance, might demand over a billion dollars and more than a decade of
development (Gaudelet et al. 2021).

With the advent of artificial intelligence (AI), innovative cross-modal
methods are ushering in new ways to synthesize and analyze complex molecular
structures, enhancing efficiency and reshaping the fields of computational
chemistry and material science. Edwards et al. (2022) proposed a novel
approach to directly translate molecules to corresponding captions and
generate molecular structures from natural language text, shown in Figure 1.
This cross-modal method heralds a future in which the design and study of
specialized molecules can be achieved through simple natural language
sentences.

(a) Molecular captioning

(b) Text-Based de novo Molecule Generation

Figure 1: Illustration of translation between molecule and description in
cross-modal molecule discovery.

Various attempts have been made to resolve these tasks. MolT5 (Edwards et al.
2022) uses SMILES (Simplified Molecular Input Line Entry System) (Weininger
1988) and molecule description respectively for masked language modeling (MLM)
(Raffel et al. 2020) as pre-training. Liu et al. (2023) pre-train models with
causal language modeling (CLM) on the sequences that blend biomedical
literature with molecular structural representations, derived from replacing
molecular entities with their SMILES representations. However, these studies
are limited by the scarcity of parallel molecule-description pairs, rendering
direct sequence-to-sequence training unfeasible. The effectiveness of
sequence-to-sequence (seq2seq) training is evident in Christofidellis et al.
(2023), where the annotated data from the downstream dataset is incorporated
for pre-training, albeit in a significantly lower ratio compared to the
unannotated data. The primary bottleneck is the annotation process itself: the
annotation of these pairs demands specialized knowledge in molecular
chemistry, rendering large-scale human annotation both expensive and
difficult.

Inspired by the great success of LLMs in natural language processing (NLP) and
related fields (Bagal et al. 2021; Frey et al. 2022; Ferruz, Schmidt, and
Höcker 2022), we propose to mitigate the low-resource difficulty by using
artificially-real data generated by LLMs. Unlike “real data”, which originates
from genuine experimental or observational sources, this “pseudo data” or
“artificially-real data” is crafted artificially. While it mirrors the format
of real data, its content does not depict actual real-world observations,
making it potentially unsuitable for direct real-world applications.

Our approach begins by creating a comprehensive pseudo dataset intended for
seq2seq pre-training. We collect 1M unlabeled molecules from PubChem and use
the in-context learning ability of LLMs to generate descriptive captions for
these molecules. To ensure the integrity and diversity of this pseudo data, we
adopt a retrieval-based one-shot prompting strategy during generation. Through
this way, we construct the first artificially-real dataset, PseudoMD-1M,
consisting of 1,020,139 pseudo molecule-description pairs.

Based on this dataset, we explore the optimal method to leverage pseudo data.
We propose two primary methods: 1) using pseudo data exclusively during pre-
training for domain adaptation, and 2) integrating pseudo data with real data
during fine-tuning as a data augmentation technique. To offer a comprehensive
evaluation, we further compile DrugBank-23, a novel dataset derived from a
different data source than existing datasets.

In summary, our contributions are as follows:

  * •

We are the first to incorporate LLMs for low-resource molecule discovery.
Using artificially-real data generated by LLMs, we are able to mitigate the
data scarcity for the tasks. We release PseudoMD-1M, the first artificially-
real dataset for cross-modal molecule discovery, which is 33$\times$ larger
than existing real datasets.

  * •

We explore the effective construction and utilization of pseudo data. We
specifically investigate two principal techniques, including using pseudo data
as domain adaptation and data augmentation. We conduct comprehensive
experiments on existing datasets, and provide our new dataset called
DrugBank-23, which adds a novel data source compared to current datasets.

  * •

Experimental results show that despite smaller model size and amount of pre-
training data, models using artificially-real data as domain adaptation
outperform all prior methods. Furthermore, our method shows continuous
improvement with increasing volumes of pseudo data, underscoring its promising
future applications.

##  2 Related Work

###  2.1 Cross-modal Molecule Discovery

With the advancement of in-silico molecule discovery methods, the field of
molecule exploration is undergoing a transformative shift away from its
resource-intensive and costly origins (Rifaioglu et al. 2019; Gaudelet et al.
2021). Edwards, Zhai, and Ji (2021) introduce a new task Text2Mol, which uses
descriptions as search queries to retrieve the target molecules. Following
this, Edwards et al. (2022) propose two innovative tasks: molecule captioning
and text-guided de novo molecule generation. These tasks aim at translating
between molecular structures and natural language texts. MolXPT (Liu et al.
2023) leverages literature annotations of molecules to construct a pre-
training dataset. Christofidellis et al. (2023) further improves the field
with multi-task learning, which combines single-domain and cross-domain
datasets for joint training. Most recently, Li et al. (2023) propose a
strategy that enables LLMs to accomplish both molecule captioning and text-
guided molecule generation tasks. Here we take one step further to construct a
large number of high-quality parallel data pairs, in response to the data
scarcity that limits the performance of the above approaches.

###  2.2 Large Language Models

LLMs have achieved significant success in natural language processing by
scaling up to billions of parameters (Radford et al. 2018; Brown et al. 2020;
Ouyang et al. 2022). Trained on vast corpora (Singhal et al. 2023), LLMs show
more general intelligence (Bubeck et al. 2023) and remarkable capabilities
such as in-context learning (Rubin, Herzig, and Berant 2022; Min et al. 2022).
They have also obtained promising performance in chemical (Bagal et al. 2021;
Frey et al. 2022), biological (Ferruz, Schmidt, and Höcker 2022) and medical
(Wang et al. 2023a) domains. Due to their great generation capability,
numerous works have relied on LLMs to generate data for various purposes,
including creating semantic textual similarity datasets (Schick and Schütze
2021), augmenting natural language inference (Liu et al. 2022), automatically
formulating instructions (Wang et al. 2022) and improving few-shot retrieval
(Dai et al. 2022). Inspired by these achievements, we aim to employ LLMs to
generate parallel data, addressing data scarcity in cross-modal molecule
discovery.

##  3 Methodology

###  3.1 Task Overview

Here we introduce two primary tasks for cross-modal molecule discovery. First
proposed by Edwards et al. (2022), the two tasks act as a bridge between
molecule discovery and NLP and can be considered as cross-modal translation
tasks.

Figure 2:  The workflow for pseudo data generation. Starting with an unlabeled
molecule represented by its Morgan Fingerprints, two stages are involved. In
stage 1, the input molecule serves as a search query to retrieve the top-k
similar molecules from a local database containing 37,898 annotated molecule-
caption pairs. In stage 2, the retrieved molecules and their captions are
integrated into a prompt. Then LLMs perform in-context learning and generate a
description for the input molecule.

#### Molecular captioning

As illustrated in Figure 1(a), Given the SMILES representation
$\mathcal{S}_{\mathcal{M}}$ of molecule $\mathcal{M}$, the task is to generate
the corresponding descriptions $\mathcal{D}_{\mathcal{M}}$.

#### Text-Based de novo Molecule Generation

As shown in Figure 1(b), given the descriptions $\mathcal{D}_{\mathcal{M}}$ of
molecules $\mathcal{M}$, the task is to generate its corresponding SMILES
$\mathcal{S}_{\mathcal{M}}$.

###  3.2 Artificially-real Data Generation

High-quality pseudo data is the foundation for further exploration. Here we
propose PseudoMD-1M, the first pseudo dataset composed of 1M parallel
molecule-description data pairs. To acquire sufficient data, we leverage a
vast number of unlabeled molecules and use LLMs to generate corresponding
descriptions. We begin by collecting 1.1 million unannotated SMILES strings of
molecules from PubChem (Kim et al. 2023). We then employ a rigorous filtering
procedure to filter out the SMILES in downstream datasets to ensure that there
is no overlap between the collected molecules and those contained in the real
datasets (Edwards, Zhai, and Ji 2021; Zeng et al. 2022). By doing so, we
ensure that no supplementary information about the molecules present in the
real datasets is accidentally incorporated, thereby maintaining the integrity
and independence of the training process. With ChatGPT API, we generate
textual descriptions that encompass key aspects such as properties and
structural features for each unannotated molecule. To improve the quality of
generated descriptions, we implement a retrieval-based prompt paradigm that
comprises two main stages as follows: Molecule Retrieval and Few-Shot
Prompting.

Figure 3:  Comparison of data quality. We use the method proposed by Edwards
et al. (2022) to evaluate the similarity between molecule-description pairs as
an estimation of the data quality. The distribution is visualized using Kernel
Distribution Estimation. A higher Text2Mol score signifies closer molecule-
description resemblance, and “Density” represents the data concentration in a
given region.  Figure 4: Different methods for utilizing pseudo data.
Traditional training employs only the real dataset for fine-tuning. The data
augmentation approach fine-tunes the model on the combined dataset with pseudo
data incorporated. In the domain adaptation method, the model is 1 initially
pre-trained on two concurrent cross-modal translation tasks using pseudo data
as domain adaptation, and 2 further trained on each task using real data.

#### Molecule Retrieval

In-context learning (Brown et al. 2020) is one of the emergent abilities of
LLMs, and the instances used in the prompts given to the LLMs play an
important role in the generation quality. As molecules with similar structures
often display corresponding characteristics (Wang et al. 2016), we retrieve
the descriptions of annotated molecules that resemble the unlabeled molecule,
using them as the few-shot instance during prompting. Specifically, we collect
37,898 annotated molecules with captions from PubChem(Kim et al. 2023), then
retrieve the molecules with top-k Tanimoto similarity (Tanimoto 1958), a
standard measure in cheminformatics. To prevent information leakage during
testing, we exclude the molecules that are contained in the real data test set
(Edwards, Zhai, and Ji 2021; Zeng et al. 2022). This process enables the
models to learn from the information embedded within the descriptions of
molecules that possess similar properties, ensuring a more tailored and
accurate representation. Figure 3 shows the estimate of the data quality,
indicating that the few-shot prompting approach (in blue) yields higher-
quality data, more closely resembling real data than without.

#### Few-Shot Prompting

Upon retrieving the top-k results for each unlabeled molecule from our local
database, we select one example using a weighted distribution, where molecules
with higher similarity have a greater chance of being chosen. This selected
example is then incorporated into the final prompt. We opt for one-shot
prompting to minimize generation costs, as expenses increase linearly with the
number of instances included in few-shot prompts. This weighted selection
method prevents repetitive selection of the same molecule as the few-shot
example, thereby improving the diversity during generation while maintaining
the similarity between the molecule to be annotated and the few-shot example.
As shown in Figure 2, the complete prompt comprises role definition, task
description, few-shot example, and output control. The role definition and
task description give LLMs the general context and enable its learned
knowledge, while the few-shot example acts like a supplementary material for
the LLMs to refer to. Then, with the output control for format clarification,
the LLMs should be able to generate the desired description.

###  3.3 Approaches to Utilize Artificially Real Data

The ways to utilize the pseudo data decide how the model will perform on real
data. We propose and explore two primary strategies to optimize the use of
pseudo data.

#### Pseudo Data as Data Augmentation

Data augmentation strategy can be roughly categorized into two kinds,
modification of existing data and generation of pseudo data. The former takes
an existing data instance and makes certain alterations to it without changing
its inherent meaning or label, such as rotation, flipping, and cropping for
images (Krizhevsky, Sutskever, and Hinton 2012), or synonym replacement for
text (Wang and Yang 2015; Wei and Zou 2019; Miao et al. 2020). This method is
more about adding variability and noise to existing data instances than
generating completely new ones. The latter, on the other hand, involves
creating new data instances that did not exist in the original dataset based
on the characteristics and distribution of the original data, which is an
efficient alternative when real data is scarce or when creating new real data
is costly or unfeasible. Existing applications include back translation for
text (Sennrich, Haddow, and Birch 2016), and GANs for images (Goodfellow et
al. 2014).

Inspired by the latter techniques, we explore the use of pseudo data as data
augmentation. As shown in Figure 4, we keep the original data in the training
set and augment them with pseudo data during fine-tuning. Using the same
method as described in Figure 3, we assess the distribution of the real
training set and the sample the augmented pseudo data based on the same
distribution, ensuring consistency in the overall dataset distribution before
and after data augmentation. We hope that this data augmentation approach
using pseudo data will expose the model to a broader range of data patterns
and scenarios, thus enhancing its ability to recognize complex patterns and
generalize its learning to unseen data.

#### Pseudo Data as Domain Adaptation

Models pre-trained on general domain might perform less ideally when it is
applied to specific domains for which they were not explicitly trained
(Beltagy, Lo, and Cohan 2019; Malte and Ratadiya 2019). In our case, the
SMILES appears as an unfamiliar symbol to such models, making the direct fine-
tuning approach less efficient. To bridge this gap, we use pseudo data as a
second pre-training stage for domain adaptation. As shown in Figure 4, we
train the model using pseudo data for two concurrent cross-modal translation
tasks: molecular captioning and text-based de novo molecule generation. Using
a direct and bidirectional seq2seq approach, this stage is intended to empower
the model to not only recognize the SMILES representation but also to grasp
the relationship between natural language and SMILES. Given that our primary
focus at this stage is not on data authenticity, pseudo data emerges as a
preferable choice, particularly because it provides a large number of parallel
data pairs for supervised seq2seq training compared to real datasets. We then
further fine-tune it on real data to refine and enhance the model’s
understanding of SMILES for further authenticity – a critical aspect for
applications like drug discovery.

##  4 Experiments

To validate the effectiveness of using pseudo data, we conduct comprehensive
experiments comparing our proposed approaches with existing methods. We
further conduct experiments to demonstrate how the balance between real data
and pseudo data could affect model performance. All the experiments are
conducted on both molecular captioning and molecule generation. The
implementation details are listed in Appendix C.

###  4.1 Settings

#### Datasets

Currently, only a few datasets with parallel molecule-description pairs exist,
including ChEBI-20 (Edwards, Zhai, and Ji 2021) and PCdes (Zeng et al. 2022),
both constructed using data from PubChem (Kim et al. 2023). To enhance
evaluation comprehensiveness, we assemble a new dataset called DrugBank-23,
based on DrugBank (Wishart et al. 2018). We experiment on all three datasets
(ChEBI-20, PCdes, and DrugBank-23). The detailed information about these
datasets is listed in Table 1.

Info | ChEBI-20 | PCdes | DrugBank-23
---|---|---|---
Train | 26,407 | 10,500 | 17,109
Validation | 3,301 | 1,500 | 3,667
Test | 3,300 | 3,000 | 3,666
$\mathcal{L}_{\text{SMILES}}$ | 81.56 | 56.47 | 54.11
$\mathcal{L}_{\text{Description}}$ | 52.88 | 72.47 | 65.04
Data source | PubChem | DrugBank
Table 1: Details about the existing datasets and ours (DrugBank-23).
$\mathcal{L}_{\text{SMILES}}$ denotes the average length of SMILES while
$\mathcal{L}_{\text{Description}}$ denotes the average word count per
description.

#### Models

We evaluate the following methods:

  * •

T5 (Raffel et al. 2020). T5 directly fine-tuned on downstream datasets.

  * •

MolT5 (Edwards et al. 2022). T5 pre-trained with MLM using SMILES and molecule
descriptions respectively, then fine-tuned on downstream datasets.

  * •

ChatGPT (Li et al. 2023). GPT-3.5-Turbo using few-shot prompting strategy. We
cite the results from the original paper on ChEBI-20, then apply the same
strategy to test on the other datasets.

  * •

MolXPT (Liu et al. 2023). GPT-2 pre-trained with CLM using abstracts of
biomedical literature where molecules are replaced with the corresponding
SMILES, then fine-tuned on downstream datasets. As the model is currently
unavailable, we cite their results on ChEBI-20.

  * •

Text&Chem T5 (Christofidellis et al. 2023). T5 pre-trained using multi-task
learning, then fine-tuned on downstream datasets.

  * •

Aug-T5 (ours). T5 fine-tuned on datasets augmented with pseudo data from
PseudoMD-1M, sampled from 1k to 512k, doubling at each step. We report the
optimal performances for each dataset. See Appendix D for details.

  * •

Ada-T5 (ours). T5 pre-trained using molecule-description pairs from
PseudoMD-1M as domain adaptation, then fine-tuned on downstream datasets.

Model | Data scale | Steps | Backbone
---|---|---|---
MolT5 | 500M | 1M | T5large
MolXPT | 8M | 200k | GPT2medium
Text&Chem T5 | 33.5M | 131k | T5base
Aug-T5 | 0 | 0 | T5small
Ada-T5 | 1M | 100k | T5small
Table 2: Pre-training details for different Models. “M” stands for million and
“k” denotes thousand.

As shown in Table 2, both our proposed methods utilize the smallest model
scale, pre-training data, and steps, while Aug-T5 requires no additional pre-
training. We first test our methods on T5small (Aug-T5/Ada-T5) and then apply
them to T5base (Aug-T5base/Ada-T5base).

Model | Parameters | ChEBI-20 | PCdes | DrugBank-23
---|---|---|---|---
BL | RG | MET | BL | RG | MET | BL | RG | MET
T5 | 800M | 0.467†∗ | 0.478†∗ | 0.586†∗ | 0.252†∗ | 0.259†∗ | 0.367†∗ | 0.272†∗ | 0.299†∗ | 0.396†∗
MolT5 | 800M | 0.508† | 0.510†∗ | 0.614† | 0.266† | 0.272† | 0.380†∗ | 0.293† | 0.317† | 0.416†
MolXPT | 350M | 0.505†∗ | 0.511†∗ | 0.626† | - | - | - | - | - | -
Text&Chem T5 | 250M | 0.542† | 0.543† | 0.648† | 0.266† | 0.274† | 0.382† | 0.280†∗ | 0.312†∗ | 0.413†∗
ChatGPT | - | 0.482†∗ | 0.450†∗ | 0.585†∗ | 0.194†∗ | 0.193†∗ | 0.315†∗ | 0.191†∗ | 0.218†∗ | 0.325†∗
Aug-T5 | 77M | 0.515 | 0.517 | 0.621 | 0.270 | 0.275 | 0.385 | 0.297 | 0.322 | 0.421
Aug-T5base | 250M | 0.516 | 0.520 | 0.620 | 0.268 | 0.272 | 0.383 | 0.294 | 0.316 | 0.416
Ada-T5 | 77M | 0.553 | 0.552 | 0.652 | 0.295 | 0.295 | 0.406 | 0.310 | 0.337 | 0.435
Ada-T5base | 250M | 0.564 | 0.562 | 0.660 | 0.295 | 0.297 | 0.409 | 0.322 | 0.346 | 0.445
Table 3: Results of different models for molecular captioning on ChEBI-20, PCdes and DrugBank-23 datasets. ${}^{\dagger}/^{*}$ denotes that Ada-T5base/Aug-T5base perform significantly better than baselines at $p\mathrm{-value}<0.01$ using t-test. The best scores are in bold. BL: BLEU-4. RG: ROUGE-2. MET: METEOR. Model | Parameters | ChEBI-20 | PCdes | DrugBank-23
---|---|---|---|---
Acc | Val | MAC | Acc | Val | MAC | Acc | Val | MAC
T5 | 800M | 0.279†∗ | 0.902†∗ | 0.823†∗ | 0.089† | 0.910†∗ | 0.698† | 0.131†∗ | 0.923†∗ | 0.682†
MolT5 | 800M | 0.311†∗ | 0.905†∗ | 0.834†∗ | 0.097† | 0.925† | 0.695† | 0.145†∗ | 0.947† | 0.686†
MolXPT | 350M | 0.215†∗ | 0.983 | 0.859†∗ | - | - | - | - | - | -
Text&Chem T5 | 250M | 0.322†∗ | 0.943†∗ | 0.901† | 0.105† | 0.849†∗ | 0.697† | 0.149† | 0.898†∗ | 0.705
ChatGPT | - | 0.139†∗ | 0.887†∗ | 0.847†∗ | 0.044†∗ | 0.867†∗ | 0.671†∗ | 0.048†∗ | 0.852†∗ | 0.665†∗
Aug-T5 | 77M | 0.305 | 0.907 | 0.877 | 0.070 | 0.892 | 0.700 | 0.141 | 0.911 | 0.685
Aug-T5base | 250M | 0.386 | 0.955 | 0.884 | 0.098 | 0.927 | 0.696 | 0.158 | 0.952 | 0.681
Ada-T5 | 77M | 0.449 | 0.967 | 0.905 | 0.135 | 0.945 | 0.725 | 0.170 | 0.955 | 0.696
Ada-T5base | 250M | 0.486 | 0.974 | 0.911 | 0.150 | 0.956 | 0.743 | 0.192 | 0.969 | 0.706
Table 4: Results of different models for molecule generation on ChEBI-20,
PCdes and DrugBank-23 datasets. ${}^{\dagger}/^{*}$ denotes that
Ada-T5base/Aug-T5base perform significantly better than baselines at
$p\mathrm{-value}<0.01$ using t-test. The best scores are in bold. Acc:
Accuracy. Val: Validity. MAC: MACCS FTS.

(a) BLEU-4

(b) ROUGE-2

(c) ROUGE-L

(d) METEOR

Figure 5: Results of molecular captioning task using different amount of
pseudo data.

(a) Accuracy

(b) Validity

(c) MACCS FTS

(d) RDK FTS

Figure 6: Results of molecule generation task using different amount of pseudo
data.

#### Metrics

Following existing studies (Edwards et al. 2022; Liu et al. 2023;
Christofidellis et al. 2023), we evaluate the results for molecular captioning
with BLEU-2, BLEU-4 (Papineni et al. 2002), ROUGE-1, ROUGE-2, ROUGE-L (Lin
2004) and METEOR (Banerjee and Lavie 2005), and BLEU-4 (Papineni et al. 2002),
Accuracy (Edwards et al. 2022), Validity (Polykovskiy et al. 2020),
Levenshtein distance (Miller, Vandome, and McBrewster 2009), MACCS-FTS (Durant
et al. 2002), RDK-FTS (Schneider, Sayle, and Landrum 2015), Morgan-FTS (Rogers
and Hahn 2010) and FCD (Preuer et al. 2018) for text-based de novo molecule
generation. Selected metrics are presented in Tables 3, 4 and Figures 5 and 6,
with comprehensive results in Appendix D.

###  4.2 Comparison with Existing Methods

#### Results on Molecular Captioning

Table 3 shows the results of different models for molecule captioning. Ada-T5
outperforms all previous methods and achieves the state-of-the-art on all
three datasets across all the metrics. Compared to the previous state-of-the-
art, Ada-T5 uses less than 3% of the pre-training data and only a third of the
model parameters, yet requires fewer training steps, demonstrating the
effectiveness and computational efficiency of high-quality pseudo data. On the
other hand, Aug-T5 outperforms T5, MolT5, ChatGPT and has comparable
performance with MolXPT and Text&Chem T5, using 9%-30% of the parameters and
requires no pre-training. This highlights the benefit from the enhanced
diversity of descriptions by incorporating pseudo data into the training set.
Meanwhile, Ada-T5base makes an extra but relatively little progress compared
to Ada-T5, indicating that although using pseudo data for domain adaptation
could also benefit from the expansion of model size like most methods, the
exploitation of pseudo data only demands a relatively small number of
parameters. In contrast, Aug-T5base mirrors the results of its smaller
version, indicating that for data augmentation, simply increasing the model
scale may not offer substantial benefits. One thing to notice is that despite
the data used to train the model is generated by ChatGPT API, both our trained
models can still beat ChatGPT across different metrics. This indicates that
although ChatGPT can accomplish the task to a certain extent, the data it
generated can still help the models achieve a more seamless transition through
pre-training from general domain to this domain.

#### Results on Text-Based Molecule Generation

Table 4 presents the results of different models for molecule generation.
Ada-T5 achieves the best performance in all three datasets across almost all
metrics, demonstrating its capability to generate high-quality SMILES. The
only exception is that the MolXPT slightly surpasses Ada-T5 by 0.009 in
ChEBI-20 dataset on the validity metric, which is calculated using RDkit to
simply check whether the string can be successfully converted to a molecule
object without errors and whether the molecule represents a realistic and
feasible chemical structure, without any comparison to the targeted SMILES and
the input descriptions. Despite this one slight superiority, MolXPT performs
significantly worse than Ada-T5 on other metrics, meaning that although it can
generate slightly more valid SMILES, it does not take into account the
designated instructions, ergo making it one step away from real-world
application.

On the other hand, Aug-T5 surpasses some existing methods in certain datasets
on specific metrics. However, its consistency falls short compared to Ada-T5.
This variability may be traced back to the construction of molecule-
description data pairs in pseudo data: the LLMS use the real SMILES are used
as the input, leaving only the description part of the pseudo data genuinely
“pseudo”. This means that when training Aug-T5 on molecule captioning, it gets
the authentic SMILES; but when training on molecule generation, it gets the
pseudo description. Consequently, the gap between the input training data
leads to the gap between the model performance on different tasks.
Furthermore, compared with the results for molecular captioning, the base
counterparts of both methods for molecule generation exhibit pronounced
enhancements, which could also attributed to the gap between the input data,
as using the ”pseudo” part as the input for molecule generation might offer
more space for improvements, especially for larger-scale models that can
better tolerate the “pseudo” data nuances.

The difference between Aug-T5 and Ada-T5 also indicates the importance of data
authenticity and the difference between real data and pseudo data: as Ada-T5
is later fine-tuned with 100% real data (in comparison with Aug-T5, which is
fine-tuned with the mix of real data and pseudo data), its misunderstandings
about SMILES during domain adaptation through pseudo data are corrected and
therefore has a better overall performance. This further stresses that using
pseudo data for direct application may not be the optimal way to exploit its
potential.

###  4.3 Effect of the amount of pseudo data

In order to further demonstrate how the amount of pseudo data could affect
model performance, we experiment on ChEBI-20, the largest and most widely used
dataset, with varying numbers of pseudo data samples $\mathcal{N}$ from 1k to
512k.

#### Results on Molecular Captioning

Figure 5 shows the results of Ada-T5 and Aug-T5 for molecular captioning with
different amounts of pseudo data. Both Ada-T5 and Aug-T5 exhibit significant
improvements when a modest amount of pseudo data is incorporated into their
training. With just 1k pseudo data, both methods can surpass T5large and
ChatGPT and achieve a comparative performance to MolT5large and MolXPT. This
phenomena is often seen in other data augmentation strategies (Wei and Zou
2019; Sennrich, Haddow, and Birch 2016), and can be attributed to the moderate
noise introduced by the pseudo data, which in turn bolsters model
generalization. As the amount of pseudo data increases, Ada-T5 and Aug-T5
exhibit different tendencies. The performance of Aug-T5 begins to decline when
the number of pseudo data samples reaches 4k, and sees a sharp drop when it
exceeds 32k. This is possibly due to the imbalance between real data and
pseudo data: As the model becomes increasingly exposed to unreal patterns from
the pseudo data, it might shift its attention away from genuine patterns.
Consequently, the real patterns are overlooked by the model that focuses on
the artificial ones. In contrast, Ada-T5 thrives with the increasing amount of
pseudo data, evidenced by the growth of overall metrics. One possible
explanation is that Ada-T5 only uses pseudo data for pre-training, with
follow-up fine-tuning using real data. Thus, the increase of pseudo data does
not twist its grasp of genuine patterns, but instead, further amplifies the
proficiency of the model during subsequent training.

#### Results on Text-Based Molecule Generation

Figure 6 shows the results of Ada-T5 and Aug-T5 for molecule generation with
different amounts of pseudo data. Ada-T5 shows the same superiority and trend
as it does in molecular captioning with more pseudo data incorporated, while
Aug-T5 displays a non-linear trend, with the optimal choice of the amount of
pseudo data significantly larger than when applying Aug-T5 for molecular
captioning. The reason might lie in the dual nature of pseudo data: it
introduces both linguistic patterns and noise. Initially, a little bit of
pseudo data bolsters model generalization by acting as a regularizer. But as
more is added, an overbundance of noise degrades the results. However, once a
critical mass of pseudo data is reached, the model starts to recognize more
subtle and broader linguistic patterns amidst the noise, which helps in
generating more accurate SMILES strings, leading to the observed spike in
performance. After this peak, the overwhelming volume of pseudo data might
reintroduce the dominance of noise, causing a decrease in performance.

The distinct behavior of Aug-T5 in molecular captioning versus molecule
generation highlights their inherent differences. Molecular captioning, being
more flexible, can buffer linguistic variations, downplaying minor gains from
pseudo data and instead more affected by noise. In contrast, molecule
generation requires recognizing specific linguistic cues from descriptions
that lead to exact structural changes in the SMILES output, making it more
receptive to the subtle intricacies but can also discern and benefit from the
subtle patterns present in pseudo data. Overall, these results indicate that
the impact of pseudo data varies, depending on its inherent nature and the
specific task at hand.

##  5 Conclusion

In this paper, we introduce a novel approach that enhances low-resource cross-
modal molecule discovery by leveraging artificially-real data generated by
LLMs. By incorporating a retrieval-based few-shot prompting strategy, we are
able to produce high-quality pseudo molecule-description pairs. To mitigate
the scarcity of data, we released two datasets: PseudoMD-1M, the first
artificially-real dataset for molecule description, and DrugBank-23, a real
molecule-description dataset constructed from a novel source. We propose to
use pseudo data for domain adaptation and for data augmentation to explore its
optimal utilization. Experiments across different datasets show that the
former can best exploit the potential of pseudo data, achieving better
performance with less parameters and training data. Furthermore, as the
performance of the model continues to benefit from the increasing amount of
pseudo data, our approach shows the great potential of pseudo data, thereby
providing a novel and promising approach for addressing low-resource challenge
in cross-modal molecule discovery.

## References

  * Anderson (2003) Anderson, A. C. 2003.  The process of structure-based drug design.  _Chemistry & biology_, 10(9): 787–797.
  * Bagal et al. (2021) Bagal, V.; Aggarwal, R.; Vinod, P.; and Priyakumar, U. D. 2021.  MolGPT: molecular generation using a transformer-decoder model.  _Journal of Chemical Information and Modeling_ , 62(9): 2064–2076.
  * Banerjee and Lavie (2005) Banerjee, S.; and Lavie, A. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , 65–72.
  * Beltagy, Lo, and Cohan (2019) Beltagy, I.; Lo, K.; and Cohan, A. 2019.  SciBERT: A Pretrained Language Model for Scientific Text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 3615–3620.
  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33: 1877–1901.
  * Bubeck et al. (2023) Bubeck, S.; Chandrasekaran, V.; Eldan, R.; Gehrke, J.; Horvitz, E.; Kamar, E.; Lee, P.; Lee, Y. T.; Li, Y.; Lundberg, S.; et al. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_.
  * Christofidellis et al. (2023) Christofidellis, D.; Giannone, G.; Born, J.; Winther, O.; Laino, T.; and Manica, M. 2023.  Unifying molecular and textual representations via multi-task language modelling.  _arXiv preprint arXiv:2301.12586_.
  * Curtarolo et al. (2013) Curtarolo, S.; Hart, G. L.; Nardelli, M. B.; Mingo, N.; Sanvito, S.; and Levy, O. 2013.  The high-throughput highway to computational materials design.  _Nature materials_ , 12(3): 191–201.
  * Cuzzucoli Crucitti et al. (2023) Cuzzucoli Crucitti, V.; Ilchev, A.; Moore, J. C.; Fowler, H. R.; Dubern, J.-F.; Sanni, O.; Xue, X.; Husband, B. K.; Dundas, A. A.; Smith, S.; et al. 2023.  Predictive Molecular Design and Structure–Property Validation of Novel Terpene-Based, Sustainably Sourced Bacterial Biofilm-Resistant Materials.  _Biomacromolecules_ , 24(2): 576–591.
  * Dai et al. (2022) Dai, Z.; Zhao, V. Y.; Ma, J.; Luan, Y.; Ni, J.; Lu, J.; Bakalov, A.; Guu, K.; Hall, K.; and Chang, M.-W. 2022.  Promptagator: Few-shot Dense Retrieval From 8 Examples.  In _The Eleventh International Conference on Learning Representations_.
  * Durant et al. (2002) Durant, J. L.; Leland, B. A.; Henry, D. R.; and Nourse, J. G. 2002.  Reoptimization of MDL keys for use in drug discovery.  _Journal of chemical information and computer sciences_ , 42(6): 1273–1280.
  * Edwards et al. (2022) Edwards, C.; Lai, T.; Ros, K.; Honke, G.; Cho, K.; and Ji, H. 2022.  Translation between Molecules and Natural Language.  In _2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022_.
  * Edwards, Zhai, and Ji (2021) Edwards, C.; Zhai, C.; and Ji, H. 2021.  Text2mol: Cross-modal molecule retrieval with natural language queries.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 595–607.
  * Ferruz, Schmidt, and Höcker (2022) Ferruz, N.; Schmidt, S.; and Höcker, B. 2022.  ProtGPT2 is a deep unsupervised language model for protein design.  _Nature communications_ , 13(1): 4348.
  * Frey et al. (2022) Frey, N.; Soklaski, R.; Axelrod, S.; Samsi, S.; Gomez-Bombarelli, R.; Coley, C.; and Gadepally, V. 2022.  Neural scaling of deep chemical models.
  * Gaudelet et al. (2021) Gaudelet, T.; Day, B.; Jamasb, A. R.; Soman, J.; Regep, C.; Liu, G.; Hayter, J. B.; Vickers, R.; Roberts, C.; Tang, J.; et al. 2021.  Utilizing graph machine learning within drug discovery and development.  _Briefings in bioinformatics_ , 22(6): bbab159.
  * Goodfellow et al. (2014) Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27.
  * Kim et al. (2023) Kim, S.; Chen, J.; Cheng, T.; Gindulyte, A.; He, J.; He, S.; Li, Q.; Shoemaker, B. A.; Thiessen, P. A.; Yu, B.; et al. 2023.  PubChem 2023 update.  _Nucleic acids research_ , 51(D1): D1373–D1380.
  * Krizhevsky, Sutskever, and Hinton (2012) Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25.
  * Li et al. (2023) Li, J.; Liu, Y.; Fan, W.; Wei, X.-Y.; Liu, H.; Tang, J.; and Li, Q. 2023.  Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective.  _arXiv preprint arXiv:2306.06615_.
  * Lin (2004) Lin, C.-Y. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , 74–81.
  * Liu et al. (2022) Liu, A.; Swayamdipta, S.; Smith, N. A.; and Choi, Y. 2022.  WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , 6826–6847.
  * Liu et al. (2023) Liu, Z.; Zhang, W.; Xia, Y.; Wu, L.; Xie, S.; Qin, T.; Zhang, M.; and Liu, T.-Y. 2023.  MolXPT: Wrapping Molecules with Text for Generative Pre-training.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , 1606–1616. Toronto, Canada: Association for Computational Linguistics.
  * Malte and Ratadiya (2019) Malte, A.; and Ratadiya, P. 2019.  Evolution of transfer learning in natural language processing.  _arXiv preprint arXiv:1910.07370_.
  * Miao et al. (2020) Miao, Z.; Li, Y.; Wang, X.; and Tan, W.-C. 2020.  Snippext: Semi-supervised opinion mining with augmented data.  In _Proceedings of The Web Conference 2020_ , 617–628.
  * Miller, Vandome, and McBrewster (2009) Miller, F. P.; Vandome, A. F.; and McBrewster, J. 2009.  Levenshtein distance: Information theory, computer science, string (computer science), string metric, damerau? Levenshtein distance, spell checker, hamming distance.
  * Min et al. (2022) Min, S.; Lewis, M.; Zettlemoyer, L.; and Hajishirzi, H. 2022.  MetaICL: Learning to Learn In Context.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2791–2809.
  * Ouyang et al. (2022) Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35: 27730–27744.
  * Papineni et al. (2002) Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , 311–318.
  * Patani and LaVoie (1996) Patani, G. A.; and LaVoie, E. J. 1996.  Bioisosterism: a rational approach in drug design.  _Chemical reviews_ , 96(8): 3147–3176.
  * Polykovskiy et al. (2020) Polykovskiy, D.; Zhebrak, A.; Sanchez-Lengeling, B.; Golovanov, S.; Tatanov, O.; Belyaev, S.; Kurbanov, R.; Artamonov, A.; Aladinskiy, V.; Veselov, M.; et al. 2020.  Molecular sets (MOSES): a benchmarking platform for molecular generation models.  _Frontiers in pharmacology_ , 11: 565644.
  * Preuer et al. (2018) Preuer, K.; Renz, P.; Unterthiner, T.; Hochreiter, S.; and Klambauer, G. 2018.  Fréchet ChemNet distance: a metric for generative models for molecules in drug discovery.  _Journal of chemical information and modeling_ , 58(9): 1736–1741.
  * Radford et al. (2018) Radford, A.; Narasimhan, K.; Salimans, T.; Sutskever, I.; et al. 2018.  Improving language understanding by generative pre-training.
  * Raffel et al. (2020) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1): 5485–5551.
  * Rifaioglu et al. (2019) Rifaioglu, A. S.; Atas, H.; Martin, M. J.; Cetin-Atalay, R.; Atalay, V.; and Doğan, T. 2019.  Recent applications of deep learning and machine intelligence on in silico drug discovery: methods, tools and databases.  _Briefings in bioinformatics_ , 20(5): 1878–1912.
  * Rogers and Hahn (2010) Rogers, D.; and Hahn, M. 2010.  Extended-connectivity fingerprints.  _Journal of chemical information and modeling_ , 50(5): 742–754.
  * Rubin, Herzig, and Berant (2022) Rubin, O.; Herzig, J.; and Berant, J. 2022.  Learning To Retrieve Prompts for In-Context Learning.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2655–2671.
  * Schick and Schütze (2021) Schick, T.; and Schütze, H. 2021.  Generating Datasets with Pretrained Language Models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 6943–6951. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics.
  * Schneider, Sayle, and Landrum (2015) Schneider, N.; Sayle, R. A.; and Landrum, G. A. 2015.  Get Your Atoms in Order An Open-Source Implementation of a Novel and Robust Molecular Canonicalization Algorithm.  _Journal of chemical information and modeling_ , 55(10): 2111–2120.
  * Sennrich, Haddow, and Birch (2016) Sennrich, R.; Haddow, B.; and Birch, A. 2016.  Improving Neural Machine Translation Models with Monolingual Data.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 86–96.
  * Singhal et al. (2023) Singhal, K.; Azizi, S.; Tu, T.; Mahdavi, S. S.; Wei, J.; Chung, H. W.; Scales, N.; Tanwani, A.; Cole-Lewis, H.; Pfohl, S.; et al. 2023.  Large language models encode clinical knowledge.  _Nature_ , 1–9.
  * Tanimoto (1958) Tanimoto, T. T. 1958.  Elementary mathematical theory of classification and prediction.
  * Wang et al. (2023a) Wang, H.; Liu, C.; Xi, N.; Qiang, Z.; Zhao, S.; Qin, B.; and Liu, T. 2023a.  HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge.  arXiv:2304.06975.
  * Wang and Yang (2015) Wang, W. Y.; and Yang, D. 2015.  That’s so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using# petpeeve tweets.  In _Proceedings of the 2015 conference on empirical methods in natural language processing_ , 2557–2563.
  * Wang et al. (2022) Wang, Y.; Kordi, Y.; Mishra, S.; Liu, A.; Smith, N. A.; Khashabi, D.; and Hajishirzi, H. 2022.  Self-instruct: Aligning language model with self generated instructions.  _arXiv preprint arXiv:2212.10560_.
  * Wang et al. (2016) Wang, Z.; Liang, L.; Yin, Z.; and Lin, J. 2016.  Improving chemical similarity ensemble approach in target prediction.  _Journal of cheminformatics_ , 8: 1–10.
  * Wang et al. (2023b) Wang, Z.; Liu, T.; Peng, H.; and Fang, Y. 2023b.  Advances in molecular design and photophysical engineering of perylene bisimide-containing polyads and multichromophores for film-based fluorescent sensors.  _The Journal of Physical Chemistry B_ , 127(4): 828–837.
  * Wei and Zou (2019) Wei, J.; and Zou, K. 2019.  EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 6382–6388.
  * Weininger (1988) Weininger, D. 1988.  SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules.  _Journal of chemical information and computer sciences_ , 28(1): 31–36.
  * Wishart et al. (2018) Wishart, D. S.; Feunang, Y. D.; Guo, A. C.; Lo, E. J.; Marcu, A.; Grant, J. R.; Sajed, T.; Johnson, D.; Li, C.; Sayeeda, Z.; et al. 2018.  DrugBank 5.0: a major update to the DrugBank database for 2018.  _Nucleic acids research_ , 46(D1): D1074–D1082.
  * Zeng et al. (2022) Zeng, Z.; Yao, Y.; Liu, Z.; and Sun, M. 2022.  A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals.  _Nature communications_ , 13(1): 862.

##  Appendix A More information about datasets

###  A.1 PseudoMD-1M

The PseudoMD-1M dataset comprises 1,020,139 molecule-description pairs. Every
molecule is represented using its Canonical SMILES notation, sourced from
PubChem via the PUG View API. On average, each description within PseudoMD-1M
contains 5.11 sentences, 106.47 words, and 165.07 tokens, as shown in Figure
7(a) and Figure 7(b). We provide five examples in Table 5. While we have
employed simple rule-based post-processing, we cannot guarantee its complete
accuracy and ensure that there remains free from errors, biases, or other
inaccuracies. Consequently, this dataset should not be employed in real-world
applications.

(a) The amount of token.

(b) The amount of word.

Figure 7: Token and word distributions in PseudoMD-1M.

###  A.2 DrugBank-23

The DrugBank-23 dataset comprises 17,109 molecule-description pairs, sourced
from the DrugBank dataset. In line with prior studies, we represent each
molecule using its Canonical SMILES notation and descriptions with fewer than
20 words are dropped. On average, each description contains 4.12 sentences,
64.86 words, and 119.45 tokens. Detailed distributions of word and token
counts can be seen in Figure 8(a) and 8(b), respectively. Representative
examples are provided in Table 6.

(a) The distribution of the token amount in DrugBank-23.

(b) The distribution of the word amount in DrugBank-23.

Figure 8: Token and word distributions in DrugBank-23.

SMILES representation of molecule |  Description
---|---
C1=CC(=CC(=C1)O)NC(=O)C(=O)NNC(=O)C2=CC=NC=C2 |  The molecule is a derivative of benzene featuring a carbonyl group attached to an amide, which is further attached to a nitrogen atom. It also contains a pyridine ring. This compound is a synthetic small molecule inhibitor of a class of enzymes known as poly(ADP-ribose) polymerases (PARPs). It has been investigated for its use as a potential therapeutic agent against cancer due to its ability to inhibit DNA repair pathways in cancer cells. Additionally, it has shown potential in treating neurodegenerative disorders and as an anti-inflammatory agent. It is a member of benzamides and pyridines.
C1CCC2=C(C1)C=CC(=C2)C3=NN=C(O3)NC(=O)C4=C(C=CC(=C4)Cl)[N+](=O)[O-] |  The molecule is a pyrimidine-based compound that contains a bicyclic ring system. It has a substituted phenyl group attached to the pyrimidine ring and a nitro group at position 4. Additionally, it has a chlorine atom substituted on the phenyl ring and an oxime group attached to the pyrimidine ring at position 3. The compound is a potent inhibitor of a specific enzyme and is used in the treatment of certain types of cancer. It is also a C-nitro compound and a nitrogen-containing heterocycle.
CC(=NNC(=S)N)CC1CC2=C(CC1(C)C)C(CCC2)(C)C |  The molecule is a thiosemicarbazone derivative with a cyclohexene ring that contains a branched methyl group and a carbonyl group. It has a role as an antiviral agent and a potential anticancer agent. It acts by inhibiting the replication of several viruses by blocking the activity of RNA-dependent RNA polymerase. It also has the potential to target tumor cells by inducing apoptosis and cell cycle arrest.
C1C=CCC2C1C(=O)N(C2=O)C3=CC=C(C=C3)C(=O)OC4=CC=C(C=C4)[N+](=O)[O-] |  The molecule is a diketopiperazine derivative obtained by condensation of the carboxylic acid group of one molecule of cyclo(leucylprolyl) with the amino group of another. It is functionally related to cyclo(leucylprolyl) and contains a nitro group. The molecule is believed to inhibit the activity of proteasomes, which are involved in the degradation of intracellular proteins, and is used in the treatment of multiple myeloma and mantle cell lymphoma.
CC1=CC(=C(N1CCOC)C)C(=O)COC(=O)C2=C(C3=CC=CC=C3O2)COC4=CC=CC=C4 |  The molecule is a synthetic compound with potential anti-cancer activity. It works by inhibiting histone deacetylase, which can lead to an increase in acetylation of histone proteins and subsequent transcriptional activation of tumor suppressor genes. In addition, it has been shown to induce apoptosis and cell cycle arrest in cancer cells. This compound has also been found to have anti-inflammatory effects by inhibiting the production of cytokines and chemokines. The compound is administered orally and is being studied for its potential in the treatment of various types of cancer.
Table 5: Five examples of PseudoMD-1M SMILES representation of molecule |  Description
---|---
C1=CC(=CC(=C1)[N+](=O)[O-])C2=NOC(=N2)CCCC(=O)O |  The molecule is a 1,2,4-oxadiazole, a C-nitro compound and a monocarboxylic acid. It is functionally related to a butyric acid.
COC1=CC=CC=C1OCC(CO)O |  The molecule is readily available and used as an expectorant. Since that time the agent has been a combination component of various prescription and non-prescription over-the-counter cough and cold products and is currently a widely available over-the-counter generic medication. Although it is principally believed that guaifenesin elicits an action to facilitate productive cough to manage chest congestion, it is not known whether the agent can reliably mitigate coughing. Regardless, on March 1, 2007, the FDA received a petition asking the FDA to notify the public that some antitussives, expectorants, decongestants, antihistamines, and cough/cold combinations are not known to be safe and effective in children under the age of 6 years. After the negotiation between FDA and major manufacturers, a voluntary transition of labels for not using guaifenesin in children under the age of 4 years was endorsed by FDA in 2008. Furthermore, there has also been contemporary research to suggest that guaifenesin possesses and is capable of demonstrating anticonvulsant and muscle relaxant effects to some degree possibly by acting as an NMDA receptor antagonist.
C(C(C(C(=O)COP(=O)([O-])[O-])O)O)C(=O)[O-] |  The molecule is 6-phospho-5-dehydro-2-deoxy-D-gluconate. It is trianion of 6-phospho-5-dehydro-2-deoxy-D-gluconic acid arising from deprotonation of the carboxy and phosphate groups major species at pH 7.3. It is an organophosphate oxoanion, a 5-oxo monocarboxylic acid anion and a hydroxy monocarboxylic acid anion. It is a conjugate base of a 6-phospho-5-dehydro-2-deoxy-D-gluconic acid.
CCC1C(C(C(C(=O)C(CC(C(C(C(C(C(=O)O1)C)OC2CC(C(C(O2)C)O)(C)OC)C)OC3C(C(CC(O3)C)[NH+](C)C)O)(C)O)C)C)O)(C)O |  The molecule is erythromycin A1+. It is an erythromycin cation that is the conjugate acid of erythromycin A, arising from protonation of the tertiary amino group on the 3,4,6-trideoxy-3-dimethylamino-beta-D-xylo-hexopyranosyl residue major species at pH 7.3. It has a role as a bacterial metabolite. It is a conjugate acid of an erythromycin A.
C(=CC(=O)[O-])C=C(C(=O)[O-])C(=O)[O-] |  The molecule is 2-Carboxylato-cis,cis-muconate3-. It is tricarboxylate anion of 2-carboxylato-cis,cis-muconic acid major species at pH 7.3. It is a conjugate base of a 2-carboxy-cis,cis-muconic acid.
Table 6: Five examples of DrugBank-23

##  Appendix B Prompt

As shown in Figure 9, we give a specific example of prompt we use during
pseudo data generation.

Figure 9: An example of prompt for pseudo data generation. The bold SMILES
refers to the unlabelled molecule for which LLMs generate description.

##  Appendix C Implementation

We conducted our experiments on Ubuntu 22.04 using A100-PCIE-40GB and
A100-SXM4-80GB with CUDA 11.7.1. Our primary dependencies are Python 3.11.4,
PyTorch 2.0.1, Transformers 4.31.0, and Numpy 1.24.3. We initialize our Ada-T5
and Aug-T5 using public T5.1.1 checkpoints.

During pre-training, we used a learning rate of 1e-3, batch size of 128,
random seed of 42, 1000 warm-up steps, 100,000 total training steps, and
weight decay of 0.1.

For fine-tuning, we adopted the configuration in the paper “Translation
between Molecules and Natural Language”, with a learning rate of 1e-3, 50,000
fine-tuning steps, weight decay of 0.1, batch size of 32, random seed of 42,
and 1000 warm-up steps.

For evaluation, we use a greedy search with a maximum generation length of 512
during generation. The evaluation metrics script is derived from the paper
“Translation between Molecules and Natural Language”.

For other hyperparameters, we relied on the default settings of the
T5ForConditionalGeneration class in Huggingface.

##  Appendix D Complete results of all experiments

###  D.1 Comparison to existing methods

We report the complete results compared to existing methods for both molecular
captioning task and molecule generation task. The results of molecular
captioning task are shown in Table 7. The results of molecule generation task
are shown in Table 8. Based on existing studies, we select comprehensive
metrics for two tasks respectively.

For the molecular captioning task, we select the following metrics : BLEU-2,
BLEU-4, ROUGE-1, ROUGE-2, ROUGE-L and METEOR. BLEU, ROUGE, and METEOR are key
metrics for evaluating machine-generated text. BLEU and ROUGE scores primarily
focus on n-gram precision, gauging how closely the generated structures align
with the reference structures. Specifically, they identify the overlap in
terms of sequential words or subwords of varying lengths. On the other hand,
METEOR captures both exact matches and paraphrases, offering a more
comprehensive view of how closely the generated content mirrors the reference.
This holistic approach allows METEOR to detect nuances often overlooked by
strictly precision-focused metrics. All the metrics are better when higher
(denoted by $\uparrow$).

For the text-based de novo molecule generation, we select the following
metrics: BLEU-4, Accuracy , Validity, Levenshtein distance, MACCS-FTS, RDK-
FTS, Morgan-FTS and FCD. The SMILES notation represents molecules as textual
sequences, allowing the use of BLEU for evaluation. Accuracy is based on exact
SMILES string matches, while the Levenshtein distance reveals structural
differences by measuring the edits required to transform one string into
another. Validity is determined by successfully converting the SMILES string
to an RDkit object without errors. Regarding fingerprint scores, we use three
methods: MACCS, RDK, and Morgan. These are evaluated using fingerprint
Tanimoto similarity (FTS). To compare the fingerprints of two molecules,
Tanimoto similarity is employed, with the average similarity reported across
the evaluation dataset. Among these metrics, lower values are preferable for
Levenshtein distance and FCD (denoted by $\downarrow$), while higher values
are better for the rest (denoted by $\uparrow$).

Model | Parameters | Dataset | BLEU-2$\uparrow$ | BLEU-4$\uparrow$ | Rouge-1$\uparrow$ | Rouge-2$\uparrow$ | Rouge-L$\uparrow$ | METEOR$\uparrow$
---|---|---|---|---|---|---|---|---
T5 | 800M | ChEBI-20 | 0.558†∗ | 0.467†∗ | 0.630†∗ | 0.478†∗ | 0.569†∗ | 0.586†∗
PCdes | 0.336†∗ | 0.252†∗ | 0.424†∗ | 0.259†∗ | 0.361†∗ | 0.367†∗
DrugBank-23 | 0.353†∗ | 0.272†∗ | 0.449†∗ | 0.299†∗ | 0.399†∗ | 0.396†∗
MolT5 | 800M | ChEBI-20 | 0.594†∗ | 0.508† | 0.654†∗ | 0.510†∗ | 0.594†∗ | 0.614†
PCdes | 0.349† | 0.266† | 0.435† | 0.272† | 0.372† | 0.380†∗
DrugBank-23 | 0.375† | 0.293† | 0.466†∗ | 0.317† | 0.414†∗ | 0.416†
MolXPT | 350M | ChEBI-20 | 0.594†∗ | 0.505†∗ | 0.660† | 0.511†∗ | 0.597† | 0.626†
Text&Chem T5 | 250M | ChEBI-20 | 0.625† | 0.542† | 0.682† | 0.543† | 0.622† | 0.648†
PCdes | 0.352† | 0.266† | 0.439† | 0.274† | 0.373† | 0.382†
DrugBank-23 | 0.366†∗ | 0.280†∗ | 0.465†∗ | 0.312†∗ | 0.412†∗ | 0.413†∗
ChatGPT | - | ChEBI-20 | 0.565†∗ | 0.482†∗ | 0.623†∗ | 0.450†∗ | 0.543†∗ | 0.585†∗
PCdes | 0.283†∗ | 0.194†∗ | 0.375†∗ | 0.193†∗ | 0.298†∗ | 0.315†∗
DrugBank-23 | 0.281†∗ | 0.191†∗ | 0.395†∗ | 0.218†∗ | 0.329†∗ | 0.325†∗
Aug-T5small | 77M | ChEBI-20 | 0.602 | 0.515 | 0.661 | 0.517 | 0.602 | 0.621
Pcdes | 0.354 | 0.270 | 0.440 | 0.275 | 0.375 | 0.385
DrugBank-23 | 0.379 | 0.297 | 0.473 | 0.322 | 0.421 | 0.421
Aug-T5base | 250M | ChEBI-20 | 0.601 | 0.517 | 0.662 | 0.521 | 0.603 | 0.621
Pcdes | 0.352 | 0.268 | 0.439 | 0.272 | 0.374 | 0.383
DrugBank-23 | 0.377 | 0.294 | 0.468 | 0.316 | 0.415 | 0.416
Ada-T5small | 77M | ChEBI-20 | 0.633 | 0.553 | 0.686 | 0.552 | 0.626 | 0.652
PCdes | 0.378 | 0.295 | 0.455 | 0.295 | 0.391 | 0.406
DrugBank-23 | 0.390 | 0.310 | 0.486 | 0.337 | 0.433 | 0.435
Ada-T5base | 250M | ChEBI-20 | 0.642 | 0.564 | 0.693 | 0.562 | 0.634 | 0.660
PCdes | 0.378 | 0.295 | 0.459 | 0.297 | 0.393 | 0.409
DrugBank-23 | 0.404 | 0.322 | 0.494 | 0.346 | 0.440 | 0.445
Table 7: Overall comparison of methods for molecular captioning task across three datasets. ${}^{\dagger}/^{*}$ denotes that Ada-T5base/Aug-T5base perform significantly better than baselines at $p\mathrm{-value}<0.01$ level using t-test. The best results are in bold. Model | Parameters | Dataset | BL$\uparrow$ | Acc$\uparrow$ | Ls$\downarrow$ | Val$\uparrow$ | MACCS$\uparrow$ | RDK$\uparrow$ | Morgan$\uparrow$ | FCD$\downarrow$
---|---|---|---|---|---|---|---|---|---|---
T5 | 800M | ChEBI-20 | 0.854†∗ | 0.279†∗ | 16.72†∗ | 0.902†∗ | 0.823†∗ | 0.731†∗ | 0.670†∗ | 1.220†∗
PCdes | 0.651† | 0.089† | 25.18† | 0.910†∗ | 0.698† | 0.551† | 0.456† | 0.980†∗
DrugBank-23 | 0.665†∗ | 0.131†∗ | 23.97† | 0.923†∗ | 0.682† | 0.571† | 0.465† | 0.825†∗
MolT5 | 800M | ChEBI-20 | 0.854†∗ | 0.311†∗ | 16.07†∗ | 0.905†∗ | 0.834†∗ | 0.746†∗ | 0.684†∗ | 1.200†∗
PCdes | 0.652† | 0.097† | 25.60† | 0.925† | 0.695† | 0.554† | 0.463† | 0.994†∗
DrugBank-23 | 0.667†∗ | 0.145†∗ | 23.87† | 0.947† | 0.686† | 0.577† | 0.468† | 0.794†∗
MolXPT | 350M | ChEBI-20 | - | 0.215†∗ | - | 0.983 | 0.859†∗ | 0.757†∗ | 0.667†∗ | 0.450†∗
Text&Chem T5 | 250M | ChEBI-20 | 0.853†∗ | 0.322†∗ | 16.87†∗ | 0.943†∗ | 0.901† | 0.816† | 0.757† | 0.050
PCdes | 0.614†∗ | 0.105† | 30.43†∗ | 0.849†∗ | 0.697† | 0.544† | 0.459† | 1.139†∗
DrugBank-23 | 0.677 | 0.149† | 23.41† | 0.898†∗ | 0.705 | 0.585† | 0.480† | 0.789†
ChatGPT | - | ChEBI-20 | 0.790†∗ | 0.139†∗ | 24.91†∗ | 0.887†∗ | 0.847†∗ | 0.708†∗ | 0.624†∗ | 0.570†∗
PCdes | 0.526†∗ | 0.044†∗ | 37.69†∗ | 0.867†∗ | 0.671†∗ | 0.470†∗ | 0.364†∗ | 1.285†∗
DrugBank-23 | 0.435†∗ | 0.048†∗ | 43.18†∗ | 0.852†∗ | 0.665†∗ | 0.491†∗ | 0.371†∗ | 1.414†∗
Aug-T5small | 77M | ChEBI-20 | 0.861 | 0.305 | 16.84 | 0.907 | 0.877 | 0.779 | 0.705 | 0.513
PCdes | 0.651 | 0.070 | 26.25 | 0.892 | 0.700 | 0.541 | 0.438 | 0.950
DrugBank-23 | 0.675 | 0.141 | 23.17 | 0.911 | 0.685 | 0.568 | 0.469 | 0.770
Aug-T5base | 250M | ChEBI-20 | 0.872 | 0.386 | 14.24 | 0.955 | 0.884 | 0.801 | 0.733 | 0.384
PCdes | 0.654 | 0.098 | 26.25 | 0.927 | 0.696 | 0.554 | 0.460 | 0.932
DrugBank-23 | 0.665 | 0.158 | 23.93 | 0.952 | 0.681 | 0.568 | 0.471 | 0.764
Ada-T5small | 77M | ChEBI-20 | 0.880 | 0.449 | 13.02 | 0.967 | 0.905 | 0.828 | 0.764 | 0.298
PCdes | 0.657 | 0.135 | 26.63 | 0.945 | 0.725 | 0.582 | 0.491 | 0.715
DrugBank-23 | 0.667 | 0.170 | 23.99 | 0.955 | 0.696 | 0.580 | 0.476 | 0.674
Ada-T5base | 250M | ChEBI-20 | 0.885 | 0.486 | 12.88 | 0.974 | 0.911 | 0.838 | 0.778 | 0.284
PCdes | 0.675 | 0.150 | 24.95 | 0.956 | 0.743 | 0.604 | 0.510 | 0.713
DrugBank-23 | 0.679 | 0.192 | 23.19 | 0.969 | 0.706 | 0.593 | 0.490 | 0.599
Table 8: Overall comparison of methods for molecule generation task across
three datasets. ${}^{\dagger}/^{*}$ denotes that Ada-T5base/Aug-T5base perform
significantly better than baselines at $p\mathrm{-value}<0.01$ level using
t-test. The best scores are in bold. BL: BLEU-4. Acc: Accuracy or exact match.
Ls: Levenshtein. Val: Validity. MACCS: MACCS FTS. RDK: RDK FTS. Morgan: Morgan
FTS.

###  D.2 Effect of the amount of pseudo data

Here we report the complete results of Ada-T5 and Aug-T5 across data sizes
ranging from 1k to 512k, using ChEBI-20, the largest and most widely used
dataset. The data are organized into four tables: Tables 10 and 9 detail
Ada-T5’s performance on molecular captioning and molecule generation, while
Tables 11 and 12 highlight Aug-T5’s performance for these tasks.

In the main content, we report the optimal performances of Aug-T5 and
Aug-T5base for each dataset, both of which use the same configuration. For
detailed data incorporation, in the molecular captioning task, we used 2k
extra pseudo data for ChEBI-20 and PCdes, and 4k for DrugBank-23. For molecule
generation, we used 32k for ChEBI-20, 2k for PCdes, and 1k for DrugBank-23.

$\mathcal{N}$ | BLEU-4$\uparrow$ | Accuracy$\uparrow$ | Levenshtein$\downarrow$ | Validity$\uparrow$ | MACCS FTS$\uparrow$ | RDk FTS$\uparrow$ | Morgan FTS$\uparrow$ | FCD$\downarrow$
---|---|---|---|---|---|---|---|---
0k | 0.752 | 0.196 | 28.82 | 0.808 | 0.827 | 0.699 | 0.628 | 1.039
1k | 0.853 | 0.320 | 17.15 | 0.928 | 0,860 | 0.764 | 0.692 | 0.431
2k | 0.857 | 0.326 | 16.80 | 0.933 | 0.862 | 0.770 | 0.697 | 0.428
4k | 0.855 | 0.339 | 16.71 | 0.931 | 0.865 | 0.771 | 0.701 | 0.432
8k | 0.858 | 0.352 | 15.82 | 0.941 | 0.869 | 0.782 | 0.709 | 0.408
16k | 0.865 | 0.349 | 16.65 | 0.944 | 0.875 | 0.783 | 0.714 | 0.409
32k | 0.870 | 0.368 | 15.47 | 0.944 | 0.885 | 0.797 | 0.727 | 0.394
64k | 0.870 | 0.388 | 15.02 | 0.957 | 0.887 | 0.802 | 0.730 | 0.370
128k | 0.878 | 0.392 | 14.07 | 0.959 | 0.892 | 0.808 | 0.738 | 0.337
256k | 0.880 | 0.414 | 14.26 | 0.967 | 0.896 | 0.814 | 0.745 | 0.314
512k | 0.881 | 0.413 | 13.87 | 0.968 | 0.900 | 0.817 | 0.747 | 0.308
Table 9: Results of Ada-T5 on molecule generation task using different volumes of pre-training pseudo data. $\mathcal{N}$ denotes the amount of pseudo data used in pre-training stage. The best scores are in bold. $\mathcal{N}$ | BL2 | BL4 | RG1 | RG2 | RGL | MET
---|---|---|---|---|---|---
0k | 0.529 | 0.434 | 0.612 | 0.453 | 0.551 | 0.556
1k | 0.592 | 0.506 | 0.657 | 0.511 | 0.596 | 0.613
2k | 0.599 | 0.513 | 0.660 | 0.517 | 0.600 | 0.620
4k | 0.603 | 0.519 | 0.663 | 0.522 | 0.603 | 0.624
8k | 0.609 | 0.525 | 0.669 | 0.528 | 0.608 | 0.628
16k | 0.608 | 0.525 | 0.666 | 0.525 | 0.606 | 0.628
32k | 0.615 | 0.533 | 0.673 | 0.534 | 0.613 | 0.635
64k | 0.619 | 0.536 | 0.674 | 0.536 | 0.615 | 0.638
128k | 0.620 | 0.537 | 0.676 | 0.537 | 0.615 | 0.640
256k | 0.623 | 0.540 | 0.679 | 0.540 | 0.618 | 0.643
512k | 0.626 | 0.543 | 0.679 | 0.542 | 0.619 | 0.645
Table 10: Results of Ada-T5 on molecular captioning task using different volumes of pre-training pseudo data. $\mathcal{N}$ denotes the amount of pseudo data used in pre-training stage. BL2: BLEU-2. BL4: BLEU-4. RG1: Rouge-1. RG2: Rouge-2. RGL: Rouge-L. MET: METEOR. The best scores are in bold. $\mathcal{N}$ | BL2 | BL4 | RG1 | RG2 | RGL | MET
---|---|---|---|---|---|---
0k | 0.529 | 0.434 | 0.612 | 0.453 | 0.551 | 0.556
1k | 0.596 | 0.509 | 0.659 | 0.513 | 0.598 | 0.618
2k | 0.601 | 0.514 | 0.661 | 0.517 | 0.602 | 0.621
4k | 0.592 | 0.504 | 0.653 | 0.505 | 0.592 | 0.614
8k | 0.579 | 0.490 | 0.648 | 0.500 | 0.588 | 0.604
16k | 0.563 | 0.471 | 0.634 | 0.482 | 0.572 | 0.590
32k | 0.551 | 0.460 | 0.633 | 0.480 | 0.572 | 0.589
64k | 0.515 | 0.417 | 0.603 | 0.442 | 0.543 | 0.548
128k | 0.478 | 0.378 | 0.576 | 0.409 | 0.516 | 0.516
256k | 0.357 | 0.262 | 0.502 | 0.330 | 0.446 | 0.430
512k | 0.313 | 0.222 | 0.475 | 0.302 | 0.420 | 0.400
Table 11: Results of Aug-T5 on molecular captioning task using different volumes of augmented pseudo data. $\mathcal{N}$ denotes the amount of additional pseudo data integrated in fine-tuning. BL2: BLEU-2. BL4: BLEU-4. RG1: Rouge-1. RG2: Rouge-2. RGL: Rouge-L. MET: METEOR. The best scores are in bold. $\mathcal{N}$ | BLEU-4$\uparrow$ | Accuracy$\uparrow$ | Levenshtein$\downarrow$ | Validity$\uparrow$ | MACCS FTS$\uparrow$ | RDk FTS$\uparrow$ | Morgan FTS$\uparrow$ | FCD$\downarrow$
---|---|---|---|---|---|---|---|---
0k | 0.752 | 0.196 | 28.82 | 0.808 | 0.827 | 0.699 | 0.628 | 1.039
1k | 0.837 | 0.203 | 19.72 | 0.902 | 0.863 | 0.746 | 0.665 | 0.587
2k | 0.840 | 0.202 | 19.31 | 0.919 | 0.862 | 0.741 | 0.658 | 0.535
4k | 0.832 | 0.208 | 20.07 | 0.912 | 0.859 | 0.743 | 0.657 | 0.554
8k | 0.829 | 0.176 | 20.48 | 0.916 | 0.856 | 0.732 | 0.643 | 0.570
16k | 0.803 | 0.150 | 23.80 | 0.903 | 0.840 | 0.704 | 0.616 | 0.721
32k | 0.861 | 0.305 | 16.84 | 0.907 | 0.877 | 0.779 | 0.705 | 0.513
64k | 0.804 | 0.200 | 23.90 | 0.870 | 0.847 | 0.724 | 0.638 | 0.911
128k | 0.790 | 0.196 | 25.03 | 0.885 | 0.847 | 0.718 | 0.633 | 0.775
256k | 0.706 | 0.123 | 34.15 | 0.882 | 0.818 | 0.661 | 0.563 | 1.023
512k | 0.625 | 0.060 | 43.89 | 0.860 | 0.776 | 0.592 | 0.492 | 1.573
Table 12: Results of Aug-T5 on molecule generation task using different
volumes of augmented pseudo data. $\mathcal{N}$ denotes the amount of
additional pseudo data integrated in fine-tuning. The best scores are in bold.

##  Appendix E Performance Analysis: Real vs. Pseudo Data Fine-tuning

To further demonstrate the necessity of our exploration of how to effectively
utilize pseudo data, we conduct a comparative experiment on ChEBI-20 for
molecular captioning, to show that the direct application of pseudo data as a
substitution of real data hardly works for real-world scenarios and cannot
exploit the full potential of pseudo data compared to our proposed methods.

Figure 10 displays the trends of BLEU-4 and Rouge-2 scores. As training data
volume grows, performance improves significantly with real data, whereas
performance with pseudo data remains stable at a relatively low level. Table
13 and 14 provide detailed results for models trained on real and pseudo data,
respectively, and all metrics are consistent with the trends observed with
BLEU-4 and Rouge-2. These divergent performance trends highlight the value of
data authenticity and the gap between real and pseudo data. While our pseudo
data cannot yet match and replace real data, incorporating it using our
methods substantially enhances model performance, underscoring the efficacy of
our techniques for exploiting the potential of pseudo data.

$\mathcal{N}$ | BL2 | BL4 | RG1 | RG2 | RGL | MET
---|---|---|---|---|---|---
1k | 0.120 | 0.059 | 0.277 | 0.119 | 0.269 | 0.161
2k | 0.130 | 0.066 | 0.320 | 0.154 | 0.285 | 0.181
4k | 0.181 | 0.107 | 0.358 | 0.190 | 0.313 | 0.243
8k | 0.286 | 0.195 | 0.445 | 0.271 | 0.396 | 0.352
16k | 0.447 | 0.345 | 0.553 | 0.382 | 0.497 | 0.481
24k | 0.506 | 0.407 | 0.597 | 0.434 | 0.540 | 0.537
26k | 0.529 | 0.434 | 0.612 | 0.453 | 0.551 | 0.556
Table 13: Performance of the model using varying amounts of real data only. $\mathcal{N}$ denotes the amount of pseudo data. BL2: BLEU-2. BL4: BLEU-4. RG1: Rouge-1. RG2: Rouge-2. RGL: Rouge-L. MET: METEOR. The best scores are in bold. $\mathcal{N}$ | BL2 | BL4 | RG1 | RG2 | RGL | MET
---|---|---|---|---|---|---
1k | 0.127 | 0.051 | 0.262 | 0.087 | 0.190 | 0.199
2k | 0.159 | 0.069 | 0.292 | 0.103 | 0.224 | 0.218
4k | 0.163 | 0.069 | 0.299 | 0.104 | 0.230 | 0.222
8k | 0.165 | 0.078 | 0.303 | 0.106 | 0.229 | 0.222
16k | 0.175 | 0.085 | 0.309 | 0.126 | 0.239 | 0.229
32k | 0.174 | 0.084 | 0.310 | 0.125 | 0.236 | 0.224
64k | 0.173 | 0.084 | 0.309 | 0.127 | 0.236 | 0.221
128k | 0.171 | 0.081 | 0.307 | 0.125 | 0.237 | 0.223
256k | 0.167 | 0.079 | 0.305 | 0.122 | 0.233 | 0.222
512k | 0.183 | 0.092 | 0.319 | 0.134 | 0.245 | 0.231
Table 14: Performance of the model using varying amounts of pseudo data only.
$\mathcal{N}$ denotes the amount of pseudo data. BL2: BLEU-2. BL4: BLEU-4.
RG1: Rouge-1. RG2: Rouge-2. RGL: Rouge-L. MET: METEOR. The best scores are in
bold.

(a) BLEU-4.

(b) Rouge-2.

Figure 10: Trends in metrics as pseudo and real data volumes increase.