# Comprehensive evaluation of no-reference image quality assessment algorithms
on authentic distortions

Domonkos Varga

###### Abstract

Objective image quality assessment deals with the prediction of digital
images’ perceptual quality. No-reference image quality assessment predicts the
quality of a given input image without any knowledge or information about its
pristine (distortion free) counterpart. Machine learning algorithms are
heavily used in no-reference image quality assessment because it is very
complicated to model the human visual system’s quality perception. Moreover,
no-reference image quality assessment algorithms are evaluated on publicly
available benchmark databases. These databases contain images with their
corresponding quality scores. In this study, we evaluate several machine
learning based NR-IQA methods and one opinion unaware method on databases
consisting of authentic distortions. Specifically, LIVE In the Wild and
KonIQ-10k databases were applied to evaluate the state-of-the-art. For machine
learning based methods, appx. 80$\%$ were used for training and the remaining
20$\%$ were used for testing. Furthermore, average PLCC, SROCC, and KROCC
values were reported over 100 random train-test splits. The statistics of
PLCC, SROCC, and KROCC values were also published using boxplots. Our
evaluation results may be helpful to obtain a clear understanding about the
status of state-of-the-art no-reference image quality assessment methods.

_K_ eywords no-reference image quality assessment

##  1 Introduction

After digital images are captured, they suffer from a wide variety of
distortions due to compression, editing, retargeting, and transmission. The
above mentioned mechanisms influence the perceived quality of digital images.
Objective image quality assessment (IQA) deals with the measurement of
perceptual image quality where image processing, computer vision, artificial
intelligence, and machine learning techniques are applied to construct
computational models to predict perceived image quality. Due to our limited
understanding of the human visual system, it is very complicated to model the
human visual system’s quality perception. Thus, machine learning methods are
heavily used to imitate the processes of such complex mechanisms of the human
visual system. Since humans are the end users of digital images, the most
reliable method to evaluate the quality of digital images is to ask humans for
quality ratings of images. Subjective IQA deals with gathering subjective
scores from human observers considering the recommendations of different
international standards, such as ITU-R BT.500-11 [1] or ITU-P.910 [2].
Subsequently, the scores are processed to determine the mean opinion score
(MOS) or the difference mean opinion score (DMOS). There are several IQA
databases publicly available which contain images labelled with MOS or DMOS.
Table 1 allows a comparison of publicly available IQA databases. The majority
of IQA data sets contains a small set of pristine (distortion free) images and
distorted images are derived from the pristine image using different
artificial distortion types and levels. On the other hand, some databases
(LIVE In the Wild [3] and KonIQ-10k [4]) consist of individual images with
authentic distortions collected from the Internet or private collections.
Objective IQA algorithms are trained and tested on these databases.

In this study, our goal is to evaluate several state-of-the-art no-reference
image quality assessment (NR-IQA) algorithms, including BLIINDS-II [5], BMPRI
[6], BRISQUE [7], CurveletQA [8], DIIVINE [9], ENIQA [10], GRAD-LOG-CP [11],
GWH-GLBP-BIQA [12], IQVG [13], MultiGAP-GPR [14], MultiGAP-SVR [14], NBIQA
[15], OG-IQA [16], ORACLE [17], PIQE [18], SCORER [19], SEER [20], SPF-IQA
[21], and SSEQ [22], on databases with authentic distortions. Excluding PIQE
[18] which is an opinion-unaware method, all algorithms are machine learning
based. Machine learning based algorithms were trained on appx. 80$\%$ of
images and tested on the remaining 20$\%$. Correlation between the predicted
and the ground-truth scores was measured by Pearson’s linear correlation
coefficient (PLCC), Spearman’s rank order correlation coefficient (SROCC), and
Kendall’s rank order correlation coefficient (KROCC). In this study, we report
on the average PLCC, SROCC, and KROCC values measured over 100 random train-
test splits. Moreover, the statistics of PLCC, SROCC, and KROCC values are
summarized with boxplots.

Table 1: Comparison of several publicly available IQA databases.  Database | Ref. images | Test images | Resolution | Distortion levels | Number of distortions
---|---|---|---|---|---
LIVE [23] | 29 | 779 | $768\times 512$ | 4-5 | 5
A57 [24] | 3 | 54 | $512\times 512$ | 6 | 3
Toyoma-MICT [25] | 14 | 168 | $768\times 512$ | 6 | 2
TID2008 [26] | 25 | 1,700 | $512\times 384$ | 4 | 17
CSIQ [27] | 30 | 866 | $512\times 512$ | 4-5 | 6
VCL-FER [28] | 23 | 552 | $683\times 512$ | 6 | 4
LIVE Multiple Distorted [29] | 15 |  | $1280\times 720$ | 3 |
TID2013 [30] | 25 | 3,000 | $512\times 384$ | 5 | 24
CID:IQ [31] | 23 | 690 | $800\times 800$ | 5 | 6
LIVE In the Wild [3] | - | 1,169 | $500\times 500$ | - | N/A
MDID [32] | 20 | 1,600 | $512\times 384$ | 4 | 5
KonIQ-10k [4] | - | 10,073 | $1024\times 768$ | - | N/A
KADID-10k [33] | 81 | 10,125 | $512\times 384$ | 5 | 25

###  1.1 Contributions

The main contribution of this study is the comprehensive evaluation of several
state-of-the-art NR-IQA algorithms (BLIINDS-II [5], BMPRI [6], BRISQUE [7],
CurveletQA [8], DIIVINE [9], ENIQA [10], GRAD-LOG-CP [11], GWH-GLBP-BIQA [12],
IQVG [13], MultiGAP-GPR [14], MultiGAP-SVR [14], NBIQA [15], OG-IQA [16],
ORACLE [17], PIQE [18], SCORER [19], SEER [20], SPF-IQA [21], and SSEQ [22])
on LIVE In the Wild and KonIQ-10k databases. Average PLCC, SROCC, and KROCC
are reported measured over 100 random train-test splits (appx. 80$\%$ for
training and 20$\%$ for testing). Moreover, the statistics of PLCC, SROCC, and
KROCC values are summarized in box plots.

##  2 Experimental results

In this section, NR-IQA algorithms are evaluated on LIVE In the Wild and
KonIQ-10k databases. As already mentioned, 80$\%$ of images was used for
training and the remaining 20$\%$ for testing. Furthermore, average PLCC,
SROCC, and KROCC values are reported measured over 100 random train-test
splits.

###  2.1 LIVE In the Wild

In this subsection, we present our results measured on LIVE In the Wild
database. Average PLCC, SROCC, and KROCC are summarized in Table 2. It can be
seen that MultiGAP [14] methods, which extracts deep features from Inception
modules of an Inception-v3 convolutional neural network, achieve the best
results. MultiGAP-SVR [14] utilizes an support vector regressor (SVR) with
Gaussian kernel function, while MultiGAP-GPR [14], [34] relies on Gaussian
process regression with rational quadratic kernel function. The statistics of
PLCC, SROCC, and KROCC values are summarized in Figures 1, 2, and 3. On each
box, the central mark indicates the median, and the bottom and top edges of
the box indicate the 25$th$ and 75$th$ percentiles, respectively. The whiskers
extend to the most extreme data points not considered outliers, and the
outliers are plotted individually using the ${}^{\prime}+^{\prime}$ symbol.

Table 2: Overall performance on LIVE In the Wild. Average PLCC, SROCC, and KROCC are reported, measured 100 random train-test splits. The best results are typed by bold, the second best results are typed by italic, the third best results are underlined.  Method | PLCC | SROCC | KROCC
---|---|---|---
BLIINDS-II [5] | 0.450 | 0.419 | 0.292
BMPRI [6] | 0.521 | 0.480 | 0.332
BRISQUE [7] | 0.503 | 0.487 | 0.333
CurveletQA [8] | 0.620 | 0.611 | 0.433
DIIVINE [9] | 0.602 | 0.579 | 0.405
ENIQA [10] | 0.578 | 0.554 | 0.386
GRAD-LOG-CP [11] | 0.579 | 0.557 | 0.391
GWH-GLBP-BIQA [12] | 0.588 | 0.560 | 0.394
IQVG [13] | 0.581 | 0.559 | 0.393
MultiGAP-SVR [14] | 0.841 | 0.813 | 0.626
MultiGAP-GPR [14]. [34] | 0.857 | 0.826 | 0.641
NBIQA [15] | 0.607 | 0.593 | 0.414
OG-IQA [16] | 0.526 | 0.497 | 0.342
ORACLE [17] | 0.622 | 0.606 | 0.427
PIQE [18] | 0.171 | 0.108 | 0.070
SCORER [19] | 0.599 | 0.590 | 0.416
SEER [20] | 0.556 | 0.547 | 0.383
SPF-IQA [21] | 0.592 | 0.563 | 0.395
SSEQ [22] | 0.469 | 0.429 | 0.295

(a) BLIINDS-II.

(b) BMPRI.

(c) BRISQUE.

(d) CurveletQA.

(e) DIIVINE.

(f) ENIQA.

Figure 1: Box plots of the measured PLCC, SROCC, and KROCC values on LIVE In
the Wild database. 100 random train-test splits.

(a) GRAD-LOG-CP.

(b) GWH-GLBP-BIQA.

(c) IQVG.

(d) MultiGAP-GPR.

(e) MultiGAP-SVR.

(f) NBIQA.

Figure 2: Box plots of the measured PLCC, SROCC, and KROCC values on LIVE In
the Wild database. 100 random train-test splits.

(a) OG-IQA.

(b) ORACLE.

(c) SCORER.

(d) SEER.

(e) SPF-IQA.

(f) SSEQ.

Figure 3: Box plots of the measured PLCC, SROCC, and KROCC values on LIVE In
the Wild database. 100 random train-test splits.

###  2.2 KonIQ-10k

In this subsection, we present our results measured on KonIQ-10k database.
Average PLCC, SROCC, and KROCC are summarized in Table 3. Similar to LIVE In
the Wild, MultiGAP [14], [34] methods achieve the best performance. The
statistics of PLCC, SROCC, and KROCC values are summarized in Figures 4, 5,
and 6. As already mentioned, on each box, the central mark indicates the
median, and the bottom and top edges of the box indicate the 25$th$ and 75$th$
percentiles, respectively. The whiskers extend to the most extreme data points
not considered outliers, and the outliers are plotted individually using the
${}^{\prime}+^{\prime}$ symbol.

Table 3: Overall performance on KonIQ-10k. Average PLCC, SROCC, and KROCC are reported, measured 100 random train-test splits. The best results are typed by bold, the second best results are typed by italic, the third best results are underlined.  Method | PLCC | SROCC | KROCC
---|---|---|---
BLIINDS-II [5] | 0.571 | 0.575 | 0.403
BMPRI [6] | 0.636 | 0.619 | 0.435
BRISQUE [7] | 0.702 | 0.676 | 0.483
CurveletQA [8] | 0.728 | 0.716 | 0.520
DIIVINE [9] | 0.709 | 0.692 | 0.498
ENIQA [10] | 0.758 | 0.744 | 0.546
GRAD-LOG-CP [11] | 0.705 | 0.698 | 0.502
GWH-GLBP-BIQA [12] | 0.726 | 0.700 | 0.508
IQVG [13] | 0.688 | 0.684 | 0.489
MultiGAP-SVR [14] | 0.915 | 0.911 | 0.732
MultiGAP-GPR [14], [34] | 0.928 | 0.925 | 0.752
NBIQA [15] | 0.770 | 0.748 | 0.549
OG-IQA [16] | 0.652 | 0.635 | 0.448
ORACLE [17] | 0.757 | 0.748 | 0.550
PIQE [18] | 0.206 | 0.245 | 0.165
SCORER [19] | 0.772 | 0.762 | 0.561
SEER [20] | 0.705 | 0.705 | 0.509
SPF-IQA [21] | 0.759 | 0.740 | 0.543
SSEQ [22] | 0.584 | 0.573 | 0.402

(a) BLIINDS-II.

(b) BMPRI.

(c) BRISQUE.

(d) CurveletQA.

(e) DIIVINE.

(f) ENIQA.

Figure 4: Box plots of the measured PLCC, SROCC, and KROCC values on KonIQ-10k
database. 100 random train-test splits.

(a) GRAD-LOG-CP.

(b) GWH-GLBP-BIQA.

(c) IQVG.

(d) MultiGAP-GPR.

(e) MultiGAP-SVR.

(f) NBIQA.

Figure 5: Box plots of the measured PLCC, SROCC, and KROCC values on KonIQ-10k
database. 100 random train-test splits.

(a) OG-IQA.

(b) ORACLE.

(c) SCORER.

(d) SEER.

(e) SPF-IQA.

(f) SSEQ.

Figure 6: Box plots of the measured PLCC, SROCC, and KROCC values on KonIQ-10k
database. 100 random train-test splits.

##  3 Conclusion

In this study, we evaluated several NR-IQA algorithms on authentic
distortions. Specifically, we utilized two publicly available benchmark
databases (LIVE In the Wild and KonIQ-10k) to evaluate the state-of-the-art in
NR-IQA. Appx. 80$\%$ of images were used for training and the remaining 20$\%$
were used for testing. Average PLCC, SROCC, and KROCC values were reported
measured over 100 random train-test splits. Moreover, the boxplots of PLCC,
SROCC, and KROCC values were also published for each examined machine learning
based method. Our evaluation results may be helpful to obtain a clear
understanding about the status of state-of-the-art no-reference image quality
assessment methods.

## References

  * [1] RECOMMENDATION ITU-R BT.  Methodology for the subjective assessment of the quality of television pictures.  2002\.
  * [2] P ITU-T RECOMMENDATION.  Subjective video quality assessment methods for multimedia applications.  International telecommunication union, 1999.
  * [3] Deepti Ghadiyaram and Alan C Bovik.  Massive online crowdsourced study of subjective and objective picture quality.  IEEE Transactions on Image Processing, 25(1):372–387, 2015.
  * [4] Hanhe Lin, Vlad Hosu, and Dietmar Saupe.  Koniq-10k: Towards an ecologically valid and large-scale iqa database.  arXiv preprint arXiv:1803.08489, 2018.
  * [5] Michele A Saad, Alan C Bovik, and Christophe Charrier.  Blind image quality assessment: A natural scene statistics approach in the dct domain.  IEEE transactions on Image Processing, 21(8):3339–3352, 2012.
  * [6] Xiongkuo Min, Guangtao Zhai, Ke Gu, Yutao Liu, and Xiaokang Yang.  Blind image quality estimation via distortion aggravation.  IEEE Transactions on Broadcasting, 64(2):508–517, 2018.
  * [7] Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik.  No-reference image quality assessment in the spatial domain.  IEEE Transactions on image processing, 21(12):4695–4708, 2012.
  * [8] Lixiong Liu, Hongping Dong, Hua Huang, and Alan C Bovik.  No-reference image quality assessment in curvelet domain.  Signal Processing: Image Communication, 29(4):494–505, 2014.
  * [9] Anush Krishna Moorthy and Alan Conrad Bovik.  Blind image quality assessment: From natural scene statistics to perceptual quality.  IEEE transactions on Image Processing, 20(12):3350–3364, 2011.
  * [10] Xiaoqiao Chen, Qingyi Zhang, Manhui Lin, Guangyi Yang, and Chu He.  No-reference color image quality assessment: from entropy to perceptual quality.  EURASIP Journal on Image and Video Processing, 2019(1):77, 2019\.
  * [11] Wufeng Xue, Xuanqin Mou, Lei Zhang, Alan C Bovik, and Xiangchu Feng.  Blind image quality assessment using joint statistics of gradient magnitude and laplacian features.  IEEE Transactions on Image Processing, 23(11):4850–4862, 2014.
  * [12] Qiaohong Li, Weisi Lin, and Yuming Fang.  No-reference quality assessment for multiply-distorted images in gradient domain.  IEEE Signal Processing Letters, 23(4):541–545, 2016.
  * [13] Zhongyi Gu, Lin Zhang, and Hongyu Li.  Learning a blind image quality index based on visual saliency guided sampling and gabor filtering.  In 2013 IEEE International Conference on Image Processing, pages 186–190. IEEE, 2013.
  * [14] Domonkos Varga.  Multi-pooled inception features for no-reference image quality assessment.  Applied Sciences, 10(6):2186, 2020.
  * [15] Fu-Zhao Ou, Yuan-Gen Wang, and Guopu Zhu.  A novel blind image quality assessment method based on refined natural scene statistics.  In 2019 IEEE International Conference on Image Processing (ICIP), pages 1004–1008. IEEE, 2019.
  * [16] Lixiong Liu, Yi Hua, Qingjie Zhao, Hua Huang, and Alan Conrad Bovik.  Blind image quality assessment by relative gradient statistics and adaboosting neural network.  Signal Processing: Image Communication, 40:1–15, 2016.
  * [17] Mariusz Oszust.  Optimized filtering with binary descriptor for blind image quality assessment.  IEEE Access, 6:42917–42929, 2018.
  * [18] N Venkatanath, D Praneeth, Maruthi Chandrasekhar Bh, Sumohana S Channappayya, and Swarup S Medasani.  Blind image quality evaluation using perception based features.  In 2015 Twenty First National Conference on Communications (NCC), pages 1–6. IEEE, 2015.
  * [19] Mariusz Oszust.  Local feature descriptor and derivative filters for blind image quality assessment.  IEEE Signal Processing Letters, 26(2):322–326, 2019.
  * [20] Mariusz Oszust.  No-reference image quality assessment with local gradient orientations.  Symmetry, 11(1):95, 2019.
  * [21] Domonkos Varga.  No-reference image quality assessment based on the fusion of statistical and perceptual features.  Journal of Imaging, 6(8):75, 2020.
  * [22] Lixiong Liu, Bao Liu, Hua Huang, and Alan Conrad Bovik.  No-reference image quality assessment based on spatial and spectral entropies.  Signal Processing: Image Communication, 29(8):856–863, 2014.
  * [23] Hamid R Sheikh, Muhammad F Sabir, and Alan C Bovik.  A statistical evaluation of recent full reference image quality assessment algorithms.  IEEE Transactions on image processing, 15(11):3440–3451, 2006.
  * [24] Damon M Chandler and Sheila S Hemami.  Vsnr: A wavelet-based visual signal-to-noise ratio for natural images.  IEEE transactions on image processing, 16(9):2284–2298, 2007.
  * [25] Yuukou Horita, Keiji Shibata, Yoshikazu Kawayoke, and ZM Parvez Sazzad.  Mict image quality evaluation database.  Online], http://mict. eng. u-toyama. ac. jp/mictdb. html, 2011.
  * [26] Nikolay Ponomarenko, Vladimir Lukin, Alexander Zelensky, Karen Egiazarian, Marco Carli, and Federica Battisti.  Tid2008-a database for evaluation of full-reference visual quality assessment metrics.  Advances of Modern Radioelectronics, 10(4):30–45, 2009.
  * [27] Eric Cooper Larson and Damon Michael Chandler.  Most apparent distortion: full-reference image quality assessment and the role of strategy.  Journal of electronic imaging, 19(1):011006, 2010.
  * [28] Anđela Zarić, Nenad Tatalović, Nikolina Brajković, Hrvoje Hlevnjak, Matej Lončarić, Emil Dumić, and Sonja Grgić.  Vcl@ fer image quality assessment database.  AUTOMATIKA: časopis za automatiku, mjerenje, elektroniku, računarstvo i komunikacije, 53(4):344–354, 2012.
  * [29] Dinesh Jayaraman, Anish Mittal, Anush K Moorthy, and Alan C Bovik.  Objective quality assessment of multiply distorted images.  In 2012 Conference record of the forty sixth asilomar conference on signals, systems and computers (ASILOMAR), pages 1693–1697. IEEE, 2012.
  * [30] Nikolay Ponomarenko, Lina Jin, Oleg Ieremeiev, Vladimir Lukin, Karen Egiazarian, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, et al.  Image database tid2013: Peculiarities, results and perspectives.  Signal Processing: Image Communication, 30:57–77, 2015.
  * [31] Xinwei Liu, Marius Pedersen, and Jon Yngve Hardeberg.  Cid: Iq–a new image quality database.  In International Conference on Image and Signal Processing, pages 193–202. Springer, 2014.
  * [32] Wen Sun, Fei Zhou, and Qingmin Liao.  Mdid: A multiply distorted image database for image quality assessment.  Pattern Recognition, 61:153–168, 2017.
  * [33] Hanhe Lin, Vlad Hosu, and Dietmar Saupe.  Kadid-10k: A large-scale artificially distorted iqa database.  In 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), pages 1–3. IEEE, 2019.
  * [34] Domonkos Varga.  Comprehensive evaluation of no-reference image quality assessment algorithms on kadid-10k database.  arXiv preprint arXiv:2010.09414, 2020.


# Interactive Molecular Discovery with Natural Language

Zheni Zeng1, Bangchen Yin1, Shipeng Wang2, Jiarui Liu2, Cheng Yang3,
Haishen Yao2, Xingzhi Sun2, Maosong Sun1, Guotong Xie2*, Zhiyuan Liu1*
1 Tsinghua University 2 PingAn Technology
3 Beijing University of Posts and Telecommunications
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Natural language is expected to be a key medium for various human-machine
interactions in the era of large language models. When it comes to the
biochemistry field, a series of tasks around molecules (e.g., property
prediction, molecule mining, etc.) are of great significance while having a
high technical threshold. Bridging the molecule expressions in natural
language and chemical language can not only hugely improve the
interpretability and reduce the operation difficulty of these tasks, but also
fuse the chemical knowledge scattered in complementary materials for a deeper
comprehension of molecules. Based on these benefits, we propose the
conversational molecular design, a novel task adopting natural language for
describing and editing target molecules. To better accomplish this task, we
design ChatMol, a knowledgeable and versatile generative pre-trained model,
enhanced by injecting experimental property information, molecular spatial
knowledge, and the associations between natural and chemical languages into
it. Several typical solutions including large language models (e.g., ChatGPT)
are evaluated, proving the challenge of conversational molecular design and
the effectiveness of our knowledge enhancement method. Case observations and
analysis are conducted to provide directions for further exploration of
natural-language interaction in molecular discovery.

##  1 Introduction

Molecular design is a fundamental task in various fields including
biochemistry and material science, and has made great progress in recent years
with the development of deep learning technology Wang et al. (2022). Existing
systems usually directly generate molecules or optimize the given molecules Du
et al. (2022), in the form of various chemical languages like the simplified
molecular-input line-entry system (SMILES) Weininger (1988) and the structural
formula. On the one hand, the expression intermediate of chemical languages
lacks readability and requires a great deal of human expertise to use. On the
other hand, the task form in current molecular design systems lacks
interactivity and does not well-integrate operations such as retrieval and
editing. Therefore, though with the assistance of deep learning methods,
molecular design is still difficult and time-consuming for researchers.

Figure 1: Conversational molecular design diagram.

Luckily, the appearance of pre-training language models (PLMs) Han et al.
(2021) further brings new possibilities for this field, especially when the
large language models (LLMs) 111https://platform.openai.com/docs/model-index-
for-researchers come out and show their great capability of providing a mass
of biochemical knowledge and executing flexible human instructions Tong and
Zhang (2023). This inspires us that we may easily communicate with natural
language processing (NLP) systems about molecular design requirements.
Meanwhile, factual findings about molecules described externally with natural
language (e.g., biochemical literature) have to be taken into consideration
when understanding molecules, because they can provide complementary
information for chemical languages which characterize molecular structures and
mechanisms internally. Therefore, it is both worthy and necessary to explore
adopting the recent NLP techniques to facilitate molecular design.

Currently, preliminary explorations have been started about transferring NLP
methods to molecule tasks, such as the self-supervised architectures for
molecule comprehension Goh et al. (2018); Rong et al. (2020); Wang et al.
(2019). As for using natural language in molecular design, several tasks are
proposed including molecule-description matching Zeng et al. (2022), molecule
captioning, and generation according to descriptions Edwards et al. (2022).
However, these tasks are in various forms while not flexible and scalable
enough, and have not maximized the benefit of using natural language as an
interaction medium for molecular manipulations.

We unify existing tasks into an innovative form and propose conversational
molecular design, in which the human manipulators freely provide molecular
information in chemical or natural language, asking for readable property
descriptions or modified molecules that meet the given requirements in multi-
turn conversations. To cope with the training and evaluating demand for
conversational molecular design, we create the conversation data based on the
molecule-description parallel datasets with the help of manual-designed rule
filters.

Conversational molecular design is still a challenging task, though PLMs can
be used to understand complicated natural-language interactive commands. To be
specific, PLMs are proven to be effective when processing natural and chemical
languages isolatedly, while conversational design requires a flexible and
synergistic comprehension of two kinds of text. Besides, chemical materials
are highly specialized and it is not enough for the models to handle molecular
design with only commonsense knowledge.

To achieve the newly proposed task, we design ChatMol, which is a
knowledgeable and versatile model based on a generative PLM that can deeply
understand and generate natural language Raffel et al. (2020). To solve the
above two challenges, ChatMol is enhanced to first bridge natural and chemical
languages smoothly to enable interactive conversation, and then to master
plenty of molecular knowledge to better design molecules.

For bridging different languages, we encourage ChatMol to process natural and
chemical languages at the same time, capturing complicated associations
between two kinds of text with minimum supervision. We first regard SMILES
strings as normal texts and conduct masked language modeling on SMILES and
biochemical literature materials separately to gain a basic comprehension of
both languages. Further, we train the model to read natural language text,
recognize chemical entities, and convert to the chemical language expressions
of these entities. To automatically construct the training data, we adopt
chemical named entity recognition tools to identify entities in the literature
and query their SMILES strings from existing records.

For injecting molecular knowledge, we introduce two types of sophisticated
while representative knowledge to assist ChatMol to handle the specialized
molecular task: experimental properties from knowledge bases (KBs), and
spatial structures from toolkit calculation. The former includes physical and
chemical properties from wet-lab experiments recorded in molecule KBs in the
form of natural language. The latter includes the chemical bond, molecular
ring and aromaticity information, and this helps the spatial structure
information reflected in the SMILES expression become more intuitive. Given
the chemical SMILES following the task prefixes, our model is trained to
generate the natural language answers constructed with the above information.

Our main contributions are as followed: (1) We propose conversational
molecular design, an innovative task adopting natural language as the basic
medium of human-machine interaction for conducting low-barrier molecular
discovery; (2) We provide ChatMol, which is a pre-trained model that can
perform well in the new task with the enhancement of molecular knowledge and
chemical-natural language associations; (3) Experiment results prove the
challenge of the task we propose and also the effectiveness of our method,
achieving better performance with much fewer pre-training steps.

##  2 Conversational Molecular Design

In this section, we clarify the definition and related settings of the newly
proposed task.

###  2.1 Task Definition

In each turn of the molecular design, both the human user and the intelligent
system may provide molecules expressed in chemical language or chemical
properties expressed in natural language. Given the conversation history
$H(\mathcal{M},\mathcal{T})$ which contains molecules
$\mathcal{M}_{1,2,...,p}$ and chemical property descriptions
$\mathcal{T}_{1,2,...,q}$, we focus on two main functions that researchers
expect the intelligent system to do:

(1) Molecule understanding: The system is required to generate a paragraph of
property descriptions $\mathcal{T}^{\prime}_{i}$ for the molecule
$\mathcal{M}_{i}$ in $H$. (2) Molecule generation: The system is also supposed
to generate a specific molecule $\mathcal{M}^{\prime}_{j}$ that meets the
requirements in $H$, which may be text descriptions or the molecules that the
target has to be similar with. Since there may be more than one molecule
matching the descriptions, the human user can iteratively replenish property
descriptions $\mathcal{T}_{j+1}$, and the system would generate a modified
molecule $\mathcal{M}^{\prime}_{j+1}$ according to the current text
descriptions $\mathcal{T}_{1,2,...,j+1}$ and the last-round result
$\mathcal{M}^{\prime}_{j}$ (with the input as “it looks like
$\mathcal{M}^{\prime}_{j}$”).

Figure 2: Knowledgeable and versatile training process for ChatMol. The black
arrows indicate the input to output direction of the task.

###  2.2 Evaluation Metrics

For traditional generation tasks, evaluation metrics such as BLEU Papineni et
al. (2002), ROUGE Lin (2004) and METEOR Banerjee and Lavie (2005) are widely
used to measure the similarity of the predicted results and the reference
answers. They can also be applied to the evaluation of molecule understanding
and generation. However, the textual similarity is not persuasive from the
perspective of the accuracy of molecules.

For molecule generation, we first judge whether the generated strings are
legal molecule SMILES with the help of the RDKit toolkit Landrum et al.
(2013). We pay extra attention to the exact match (hit@1) and hit@3 accuracy.
Besides, molecular similarity measuring methods such as various fingerprint
Tanimoto similarity (FTS) metrics Tanimoto (1958); Schneider et al. (2015);
Durant et al. (2002) are also persuasive, since they detect similar functional
substructures in the molecules and can indicate the similarity of properties.

###  2.3 Model and Data Preparation

Since the features of SMILES and natural language are quite different, we
prepare two sets of encoder and decoder to separately process $\mathcal{M}$
and $\mathcal{T}$. Each set is set as a common sequence-to-sequence (seq2seq)
framework and initialized with T5 Raffel et al. (2020), a widely-adopted
seq2seq PLM.

As for the data preparation, the corresponding ${<}\mathcal{M},\mathcal{T}{>}$
pairs can be acquired from the chemical KBs, in which brief text introductions
of substances are provided. Information leakage may happen due to the
appearance of molecule names in the text. To alleviate the problem, synonyms
of the target molecule are replaced by general referential phrases such as
“the molecule”.

To conduct the multi-turn molecule generation, we create a new dataset ChEBL-
dia based on ChEBL-20. In the original dataset, $\mathcal{T}$ used to describe
the molecule $\mathcal{M}$ usually contains multiple sentences, in the order
of description from fine structures to overall properties. To get the multi-
turn text descriptions, we split $\mathcal{T}$ into sentences and reverse
their order to get $\mathcal{S}_{1,2,...,n}$ in which the amount of detailed
information increases sequentially. For the $k_{th}$ turn, we have
$\mathcal{T}_{k}=\\{\mathcal{S}_{1},\mathcal{S}_{2},...,\mathcal{S}_{k}\\}$.
To get the molecule intermediate results, we adopt the pre-trained
Molt5-caption2smiles-large model Edwards et al. (2022) to automatically
generate 5 candidates $\mathcal{M}_{k1,k2,...,k5}$ for the given
$\mathcal{T}_{k}$, and randomly pick one as the molecule $\mathcal{M}_{k}$
expected to be generated, which has RDK fingerprint similarity Landrum et al.
(2013) with the final answer $\mathcal{M}_{n}$ higher than 0.5 while lower
than 1 (i.e., to avoid information leakage).

Additionally, we filter out those items that contain only one turn of
conversation, and delete those with ’-’ in the sentences to avoid the
appearance of the standard chemical nomenclature (e.g., IUPAC nomenclature) to
directly reveal the answer. Besides, we randomly remain a few items that the
intermediate molecules have low similarities with the final answers to keep
the variety and help enhance the model’s robustness.

##  3 Methodology

In this section, we present the knowledgeable and versatile training method of
ChatMol which is designed to better finish the conversational molecular design
task. The overall process is shown in Figure 2. We mix up all the data with
different task prefixes to conduct multi-task pre-training, and apply the
plugins mentioned in molecule mapping correlation in fine-tuning.

###  3.1 Briding Different Languages

Conducting the popular pre-training paradigm masked language modeling (MLM)
ensures the basic comprehension and generation capability isolated on natural
and chemical languages. However, to process the multi-modal conversational
molecular design data, it is necessary for our model to capture the
associations and convert between the two languages. We apply the following two
strategies to construct more mapping data for our model.

Molecule Mapping Correlation. We recognize the molecular entities in the
literature corpus with the SciSpacy tool Neumann et al. (2019) and then query
for their SMILES from PubChem Kim et al. (2016). Given the natural language
segments, the molecule generation model is required to generate SMILES
expressions for all the molecules that appeared in it sequentially.
Conversely, the molecule understanding model is expected to generate the
standard names of the given molecules. In this way, we help ChatMol build the
parallel association with the least supervision.

Notice that the toolkit and KB can also play a role as plugins during the
downstream inferencing, we provide the automatically-annotated entity SMILES
strings as the prompts given text descriptions for molecule generation. To
avoid information leakage, we forcefully remove those SMILES strings in the
prompts that are the same as molecules in the answers.

Dual Augmentation. Considering that the molecule understanding (molecule-to-
text generation) and molecule-to-text generation are a pair of mutual tasks,
while existing molecule SMILES strings are much more than molecular property
descriptions in natural language, we can take the idea of dual learning He et
al. (2016); Xia et al. (2017), a common mechanism in neural machine
translation, to alleviate the challenge of lacking parallel data.
Specifically, the fine-tuned molecule understanding model can generate
augmented text descriptions for any given molecule, which can feedback on the
molecule generation training.

###  3.2 Injecting Molecular Knowledge

To get an in-depth comprehension of the given chemical language expressions
and to generate more reasonable and informative natural language descriptions,
our model is injected with two types of molecular knowledge.

Experimental Property Knowledge. We expect the model to understand the
molecular structures and to master properties explicitly described in the
natural language. We collect 15 types of physical and chemical properties
determined experimentally from PubChem database Kim et al. (2016), including
solubility, color, corrosivity and so on. These properties can be directly
described in natural language and provide supervision signals for molecule
understanding.

Spatial Structure Knowledge. Spatial information about molecules is essential
for understanding molecular properties. SMILES expressions do not directly
express the topological structure of molecules in the language model. To cope
with the demand of understanding molecule structures, we introduce spatial-
related pre-training tasks. We use the RDKit toolkit Landrum et al. (2013) to
get the spatial structure of the input molecule. For the specific atom, the
model is required to recognize the other atoms connected to it, as well as its
aromaticity and ring formation information.

##  4 Experiment

In this section, we first introduce the basic settings including datasets and
baseline models. Next, we introduce the molecule understanding and generation
experiments and the corresponding ablation study. To evaluate the model
capability more comprehensively, we also provide the case analysis.

Model | Training Steps | Dataset |  BL-2↑ |  BL-4↑ |  RG-1↑ |  RG-2↑ |  RG-L↑ |  MET↑
---|---|---|---|---|---|---|---|---
ChatGPT | - | ChEBI | 0.226 | 0.121 | 0.311 | 0.117 | 0.232 | 0.292
PCdes | 0.144 | 0.055 | 0.244 | 0.062 | 0.158 | 0.233
KV-PLM | 0 | ChEBI | 0.406 | 0.319 | 0.506 | 0.323 | 0.430 | 0.506
PCdes | 0.235 | 0.145 | 0.331 | 0.143 | 0.250 | 0.323
T5 | 0 | ChEBI | 0.630 | 0.553 | 0.665 | 0.520 | 0.604 | 0.632
PCdes | 0.374 | 0.289 | 0.440 | 0.275 | 0.377 | 0.393
MolT5 | 1 million | ChEBI | 0.626 | 0.549 | 0.661 | 0.517 | 0.601 | 0.628
PCdes | 0.343 | 0.262 | 0.433 | 0.264 | 0.368 | 0.369
ChatMol | 6,000 | ChEBI | 0.647 | 0.573 | 0.678 | 0.538 | 0.618 | 0.649
PCdes | 0.396 | 0.312 | 0.458 | 0.295 | 0.395 | 0.414
Table 1: Molecule understanding experiment results. BL: BLEU. RG: ROUGE. MET: METEOR. We display the average results under 5 different random seeds.  Model | Training Steps |  EM↑ |  hit@3↑ |  BL↑ |  Leven↓ |  RDK↑ |  MAC↑ |  Morgan↑
---|---|---|---|---|---|---|---|---
ChatGPT | - | 0.009 | 0.009 | 0.405 | 56.27 | 0.283 | 0.483 | 0.184
KV-PLM | 0 | 0.111 | 0.223 | 0.654 | 34.14 | 0.532 | 0.691 | 0.418
T5 | 0 | 0.052 | 0.078 | 0.600 | 40.71 | 0.513 | 0.672 | 0.419
MolT5 | 1 million | 0.061 | 0.085 | 0.605 | 40.47 | 0.517 | 0.667 | 0.422
ChatMol | 6,000 | 0.084 | 0.117 | 0.626 | 38.35 | 0.579 | 0.716 | 0.485
ChatMol+ | 6,000 | 0.140 | 0.183 | 0.712 | 29.89 | 0.649 | 0.764 | 0.551
Table 2: Molecule generation experiment results. EM: exact matched. Leven:
Levenshtein. RDK: RDK FTS. MAC: MACCS FTS. Morgan: Morgan FTS. We display the
average results under 5 different random seeds.

###  4.1 Basic Settings

For molecule understanding, there are few datasets focusing on mutual
generation between text and molecule, such as PCdes Zeng et al. (2022) and
ChEBI-20 Edwards et al. (2022). We adopt both as our fine-tuning and
evaluation datasets.There are 26,407 train, 3,301 validation, and 3,300 test
pieces of molecule-description pairs in ChEBI-20. Each piece of description
has 43.4 words and 3.3 sentences on average. Correspondingly, there are 10,500
train, 1,500 validation, and 3,000 test pieces in PCdes. Descriptions are
longer, with 62.1 words and 4.3 sentences on average.

For molecule generation, we evaluate on our newly-proposed ChEBI-dia
conversation dataset. We have 7,361 multi-turn dialogs for training, 1,369 for
validation and 1,311 for test. There are altogether 7,626 2-turn dialogs,
4,536 3-turn dialogs, 1,363 4-turn dialogs, 283 5-turn dialogs and 20 even
longer dialogs. The sentence characteristics are not much different from the
original ChEBI-20 dataset.

We evaluate the following models:

ChatGPT. This refers to the ‘gpt-3.5-turbo‘ model in the OpenAI series and is
one of the representative LLMs recently. For molecule understanding, we
provide 3 randomly-picked examples to tell the exact language style of the
specific datasets in the prompt. For molecule generation, we only provide 1
instance to constrain the output format of the model.

KV-PLM. We adopt the model checkpoint provided in the original paper Zeng et
al. (2022), which is a versatile model bridging natural and chemical languages
while based on SciBERT Beltagy et al. (2019), a popular PLM in the biomedical
domain. Since KV-PLM is an encoder model, we manage to choose alternative
retrieval solutions to compare the capabilities of the generation task as
fairly as possible. The model is conducted contrastive learning matching
molecules and descriptions. It retrieves the best-matched molecules among
train, validation, and test sets for molecule generation, and retrieves
sentences from the train set to compose the description paragraphs for
molecule understanding.

T5. This is the backbone of our model. We directly fine-tune T5 on the
downstream datasets.

MolT5. We adopt the model checkpoint provided in the original paper Edwards et
al. (2022) which is initialized with T5 checkpoint and then pre-trained by MLM
on molecules and texts with 1 million steps (batch size as $256$).

ChatMol(ours). We initialize ChatMol with T5 and conduct multi-task pre-
training. For the molecule understanding version, we adopt MLM, mapping
correlation, experimental property and spatial structure tasks and pre-train
altogether 6,000 steps (batch size as $256$). For the molecule generation
version, we adopt MLM and mapping correlation learning tasks for also
altogether 6,000 steps. Notice that the molecule generation version is first
tuned on the augmented data produced by the opposite model, and then fine-
tuned on ChEBI-dia.

ChatMol+(ours). This is the same model as above, while we allow the plugin use
of recognizing entities and querying their SMILES with toolkits and KBs in the
molecule generation task, as introduced in Section 3.1. The prompted SMILES
can provide mapping knowledge between natural and chemical languages in a
targeted manner.

Details for training settings are introduced in Appendix C.

Molecule Understanding
---
<INPUT> You are now a chemical assistant. I will provide you with a mole-
cule SMILES string, and you have to return a paragraph of proterty descrip-
tions for this molecule. For example:
Molecule: CCCC[Si](Cl)(Cl)Cl
Description: it appears as a colorless liquid with a pungent odor. Flash point
126Â°F. Corrosive to metals and skin. Used to make various silicon contain-
ing compounds…
Now the Molecule is: C(C(=O)O)I. What is the Description?
<OUTPUT> The molecule is an organic compound that belongs to the class of
carboxylic acid derivatives. Appears as a colorless to pale yellow liquid with
a pungent odor and can be synthesized by treating iodine with acetic anhy-
dride. It is known to be corrosive to metals and can cause severe irritation…
Molecule Generation
<INPUT> You are now a chemical assistant. I will provide you with a para-
graph of property descriptions, and you have to return a SMILES string of a
molecule that meet the requirements. For example:
Description: It is an acetate ester, a cyclic terpene ketone, a member of
furans, a limonoid and a tetracyclic triterpenoid.
Molecule: CC(=O)O[C@@H]1C[C@@H]2[C@](C=CC(=O)C2(C)C)([C@@H]
3[C@@]1(C4=CC[C@H]([C@@]4(CC3)C)[C@@H]5CC(=O)OC5)C)C
Now the Description is: It has a role as a food acidity regulator and a
fundamental metabolite. It is an intermediate metabolite in the citric acid
cycle. It looks like C(C(C(=O)O)C(=O)O)C(=O)O. What is the Molecule?
<OUTPUT> The molecule is: C(CC(=O)O)C(C(=O)O)C(=O)O.
Table 3: Examples of ChatGPT applying to conversational molecular design. Understanding | BL-2 | BL-4 | RG-L
---|---|---|---
Ours | 0.647 | 0.573 | 0.618
w/o property | 0.641 | 0.565 | 0.612
w/o spatial | 0.640 | 0.565 | 0.613
w/o mapping | 0.632 | 0.556 | 0.607
w/o all | 0.630 | 0.553 | 0.604
Generation | EM | BL | RDK
Ours | 0.140 | 0.712 | 0.649
w/o conversation | 0.129 | 0.710 | 0.640
w/o augmentation | 0.136 | 0.696 | 0.650
w/o mapping | 0.130 | 0.681 | 0.633
w/o prompting | 0.084 | 0.626 | 0.579
w/o all | 0.052 | 0.600 | 0.513
Table 4: Ablation study results.

###  4.2 Result Analysis

Results for the main experiment are shown in Table 1 and Table 2. Our model
has achieved all the best performances under different settings and metrics.
Details for the intermediate-turn results of molecule generation are
introduced in Appendix B.

Molecule Understanding Results. For molecule understanding, our method
generates reasonable property descriptions for the given molecules, and
achieves comprehensive improvement on both ChEBI-20 and PCdes in all the
metrics. As shown in Table 3, the LLM baseline ChatGPT has mastered a part of
chemical knowledge and owns a strong natural language generation capability,
and therefore can provide property descriptions in a similar style with the
given 3 instances. However, it also makes some mistakes (e.g., Iodoacetic acid
is not liquid), and the evaluation scores are not comparable with the tuned
models, proving the challenge of this task. As for the smaller models, KV-PLM
shows no advantages in description generation, and the reason may be that its
encoder architecture limits the range of descriptions that the model can
obtain. The MLM method-dominated MolT5 adapts worse on PCdes compared with
ChEBI-20 the original paper provides, indicating that the over-pre-training
may hurt the model generalization capability.

Molecule Generation Results. For molecule generation, our method generates
obviously more similar results with ground truth molecules than baselines. The
LLM baseline ChatGPT possesses certain molecular knowledge and can always
return valid molecules, while the mastered molecules are limited and therefore
can not reach a high accuracy under the 1-shot setting. KV-PLM, in contrast,
applies the retrieval setting and finds out the best-matched molecules from
the given pools, therefore can get higher hit scores, while does not really
fit the open scenarios. For the T5-based models, MolT5 shows marginal
improvement while not better than ChatMol. Further, augmented by the SMILES
obtained from the plugins, our model achieves a jump in performance, proving
the importance of bridging different languages and the efficiency of pre-
trained models using reliable tools.

Ablation study. We remove our knowledgeable strategies sequentially to show
their effectiveness. As shown in Table 4, w/o property, w/o spatial, and w/o
mapping refer to the removal of experimental property prediction, spatial
structure prediction, and molecule mapping correlation in the multi-task pre-
training; w/o conversation refers to the removal of molecules in the
conversation history; w/o augmentation refers to removing the dual
augmentation training; w/o prompting refers to removing the SMILES prompts
obtained from plugins.

As we can see, all the non-full versions of ChatMol perform worse on both
tasks, proving the effectiveness of our method. Especially the mapping
correlation training and the SMILES prompting show significantly lower scores
when being removed, showing that the capability of bridging versatile
subdomains is essential to accomplish the conversational molecular design. The
w/o conversation version proves that the iterative modification form is more
reasonable than directly providing a whole paragraph of text requirements.

Figure 3: Case study for conversational molecular design. The correct
molecules and properties are marked in green.

###  4.3 Case Study

We provide several generation cases from the ChEBI-dia test set in Figure 3.
Both the intermediate results for the multi-turn input text descriptions and
the generated property descriptions given the ground-truth molecules are
shown.

As we can see, for the easy case (line 1) that contains clear structural
descriptions, all models provide high-quality molecule outputs from the first
turn, and modify the molecule with the hydrochloride group according to the
supplemented second turn information. For the comparably difficult case (line
2) that provides more general and vague descriptions, baseline models fail to
generate correct molecules and precise descriptions, while ChatMol+ eventually
succeeds even when it makes a mistake in the first turn (which is not a valid
molecule). ChatMol also grasps some key required substructures (e.g.,
piperidine, tertiary alcohol, monochlorobenzene) and generates a quite similar
molecule, and meanwhile provides much more concise descriptions compared with
baseline models. Nevertheless, the models trained in a targeted manner (MolT5
and our models) have overall better performances than T5 in many cases, as
shown in line 3.

Imagine the future where we can apply the technology of conversational
molecular design for drug discovery, describing our command for structures and
functions and obtaining possible molecules from the model outputs. Besides,
readable descriptions can be generated when chemical researchers or learners
input any unfamiliar molecules just as we do in the case study. Currently, the
model and the data scales are both small, and the modifications and
descriptions are also not so granular. It is possible to get stronger
conversational molecular design systems that apply larger models with more
fine-grained training data, and have powerful knowledge memory to do more
reliable predictions.

##  5 Related Work

###  5.1 Knowledgeable Models

Though the deep learning models have the ability to absorb commonsense
knowledge from large-scale training data in an unsupervised manner, directly
injecting knowledge into models still shows its effectiveness. Specific to the
biochemical domain, there are plenty of PLMs that are transferred from the
general domain by simply replacing their corpus Wang et al. (2021), while
knowledgeable training and inferencing methods still exist. There are two
mainstream approaches: (1) Explicitly encode the knowledge to assist the
models. Extra modules are usually targeting designed for the downstream task.
For instance, related tags can be acquired from knowledge graphs (KGs) and
separately encoded to get knowledge fusion attention for the chemical-protein
interaction extraction Sun et al. (2020). KGs information can also be
incorporated via a hierarchical graph representation to modify the PLM
contextualized embeddings for biomedical entity/relation/event extraction
Huang et al. (2020); Lai et al. (2021). (2) Implicitly teach the knowledge
with special pre-training tasks. The knowledge items can help prepare data for
multi-task training with methods such as distant supervision. Based on KG
items, a binary classification task is added to predict whether a triplet
exists or not Hao et al. (2020), and entity detection and linking tasks are
conducted for similar purposes Yuan et al. (2021).

When it comes to SMILES strings processing, supplementary knowledge especially
spatial knowledge is also emphasized since SMILES cannot fully represent the
topological information of molecules Zhang et al. (2021). Graph
representations Elton et al. (2019) are popular with molecular deep learning
research since they do not have to consider rotation, permutation and other
issues additionally compared with other methods such as atom coordinates. Some
graph-based methods also incorporate 3D spatial information Danel et al.
(2020), using the universal force field(UFF) to generate the conformations.
GEM Fang et al. (2022) explores the effect of using different force field
constraints(e.g. MMFF Halgren (1996) and DFT Parr (1983)) on the quality of
the generated coordinates.

###  5.2 Models Bridging Versatile Subdomains

There exist countless cross-modal methods, especially in the image-text field.
Pre-trained models such as CLIP Radford et al. (2021) and Unimo Li et al.
(2021) have brought the fusion of the two modals to a new level. With similar
ideas, we can also bridge data from various subdomains that have natural gaps.

To process different forms of biochemical data, one solution is to convert
them into the same modal and form. Goh et al. Goh et al. (2018) unify the
molecular graph data and SMILES data to molecular structure images as the
model input. KV-PLM Zeng et al. (2022) is the first research trying to fuse
SMILES strings and natural language descriptions together to improve the
knowledge reserve and chemical comprehension of the model. MolT5 Edwards et
al. (2022) further develops the idea to conduct translation between molecules
and text descriptions by a generative model. Su et, al. Su et al. (2022)
propose a cross-modal graph-based model linking molecular and natural
language, while it ignores the flexibility of natural language descriptions.
Overall, bridging chemical and natural language is an innovative scenario that
has not been fully explored.

##  6 Conclusion

In this paper, we propose conversational molecular design, which is a novel
interactive paradigm adopting natural language for describing and editing
target molecules. Two specific tasks, molecule understanding and molecule
generation, are explored, and the dataset ChEBI-dia is made. We present
ChatMol, a knowledgeable generative model bridging chemical and natural
language descriptions for molecules. By injecting molecular knowledge and
bridging different languages, our model is further enhanced and proven to be
more effective than existing methods with significantly less training cost.
This technique promises to drive a new paradigm in AI-assisted molecular
design.

## Code and Data Availability

Codes and data are provided in https://github.com/Ellenzzn/ChatMol/tree/main.

## Limitations

Though proved to be effective and efficient, our method is temporarily not
validated on a larger scale of model parameters and training data. In fact,
the large version of our baseline (MolT5) is reported to be much more powerful
than the small and base versions, and this may indicate that various types of
knowledge that we have discussed can be automatically learned and mastered
once the computing and data resources are plenty.

Better evaluation methods are also urgently needed. As we have observed, some
descriptions are correct while not recorded in the ground truth texts, and
conversely, a paragraph of descriptions might not only correspond to a single
molecule. Therefore, it is hardly possible to thoroughly and reasonably
evaluate model performance currently.

## Ethics Statement

The authors declare that they have no conflict of interest. This article does
not contain any studies with animals or human beings performed by any of the
authors.

## References

  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In _Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , pages 65–72.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  Scibert: A pretrained language model for scientific text.  In _Proceedings of the 2019 EMNLP-IJCNLP_ , pages 3615–3620.
  * Danel et al. (2020) Tomasz Danel, Przemysław Spurek, Jacek Tabor, Marek Śmieja, Łukasz Struski, Agnieszka Słowik, and Łukasz Maziarka. 2020.  Spatial graph convolutional networks.  In _International Conference on Neural Information Processing_ , pages 668–675. Springer.
  * Du et al. (2022) Yuanqi Du, Tianfan Fu, Jimeng Sun, and Shengchao Liu. 2022.  Molgensurvey: A systematic survey in machine learning models for molecule design.  _arXiv preprint arXiv:2203.14500_.
  * Durant et al. (2002) Joseph L Durant, Burton A Leland, Douglas R Henry, and James G Nourse. 2002.  Reoptimization of mdl keys for use in drug discovery.  _Journal of chemical information and computer sciences_ , 42(6):1273–1280.
  * Edwards et al. (2022) Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, and Heng Ji. 2022.  Translation between molecules and natural language.  _arXiv preprint_.
  * Elton et al. (2019) Daniel C Elton, Zois Boukouvalas, Mark D Fuge, and Peter W Chung. 2019.  Deep learning for molecular design—a review of the state of the art.  _Molecular Systems Design & Engineering_, 4(4):828–849.
  * Fang et al. (2022) Xiaomin Fang, Lihang Liu, Jieqiong Lei, Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. 2022.  Geometry-enhanced molecular representation learning for property prediction.  _Nature Machine Intelligence_ , 4(2):127–134.
  * Goh et al. (2018) Garrett B Goh, Charles Siegel, Abhinav Vishnu, and Nathan Hodas. 2018.  Using rule-based labels for weak supervised learning: a chemnet for transferable chemical property prediction.  In _Proceedings of the 24th ACM SIGKDD_ , pages 302–310.
  * Halgren (1996) Thomas A Halgren. 1996.  Merck molecular force field. i. basis, form, scope, parameterization, and performance of mmff94.  _Journal of computational chemistry_ , 17(5-6):490–519.
  * Han et al. (2021) Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, et al. 2021.  Pre-trained models: Past, present and future.  _AI Open_ , 2:225–250.
  * Hao et al. (2020) Boran Hao, Henghui Zhu, and Ioannis C Paschalidis. 2020.  Enhancing clinical bert embedding using a biomedical knowledge base.  In _28th International Conference on Computational Linguistics_.
  * He et al. (2016) Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016.  Dual learning for machine translation.  _Advances in NeurIPS_ , 29.
  * Huang et al. (2020) Kung-Hsiang Huang, Mu Yang, and Nanyun Peng. 2020.  Biomedical event extraction with hierarchical knowledge graphs.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1277–1285.
  * Kim et al. (2016) Sunghwan Kim, Paul A Thiessen, Evan E Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He, Siqian He, Benjamin A Shoemaker, et al. 2016.  Pubchem substance and compound databases.  _Nucleic acids research_ , 44(D1):D1202–D1213.
  * Lai et al. (2021) Tuan Lai, Heng Ji, ChengXiang Zhai, and Quan Hung Tran. 2021.  Joint biomedical entity and relation extraction with knowledge-enhanced collective inference.  _arXiv preprint arXiv:2105.13456_.
  * Landrum et al. (2013) Greg Landrum et al. 2013.  Rdkit: A software suite for cheminformatics, computational chemistry, and predictive modeling.  _Greg Landrum_.
  * Li et al. (2021) Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2021.  Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning.  In _Proceedings of the 59th ACL-IJCNLP_ , pages 2592–2607.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S Weld. 2020.  S2orc: The semantic scholar open research corpus.  In _Proceedings of the 58th ACL_ , pages 4969–4983.
  * Loper and Bird (2002) Edward Loper and Steven Bird. 2002.  Nltk: The natural language toolkit.  In _Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics_ , pages 63–70.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Neumann et al. (2019) Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019.  Scispacy: Fast and robust models for biomedical natural language processing.  In _Proceedings of the 18th BioNLP Workshop and Shared Task_ , pages 319–327.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th ACL_ , pages 311–318.
  * Parr (1983) Robert G Parr. 1983.  Density functional theory.  _Annual Review of Physical Chemistry_ , 34(1):631–656.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in NeurIPS_ , 32.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th ICML_ , pages 8748–8763. PMLR.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rong et al. (2020) Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. 2020.  Self-supervised graph transformer on large-scale molecular data.  _Advances in NeurIPS_ , 33:12559–12571.
  * Schneider et al. (2015) Nadine Schneider, Roger A Sayle, and Gregory A Landrum. 2015.  Get your atoms in order: An open-source implementation of a novel and robust molecular canonicalization algorithm.  _Journal of chemical information and modeling_ , 55(10):2111–2120.
  * Su et al. (2022) Bing Su, Dazhao Du, Zhao Yang, Yujie Zhou, Jiangmeng Li, Anyi Rao, Hao Sun, Zhiwu Lu, and Ji-Rong Wen. 2022.  A molecular multimodal foundation model associating molecule graphs with natural language.  _arXiv preprint_.
  * Sun et al. (2020) Cong Sun, Zhihao Yang, Leilei Su, Lei Wang, Yin Zhang, Hongfei Lin, and Jian Wang. 2020.  Chemical–protein interaction extraction via gaussian probability distribution and external biomedical knowledge.  _Bioinformatics_ , 36(15):4323–4330.
  * Tanimoto (1958) Taffee T Tanimoto. 1958.  Elementary mathematical theory of classification and prediction.
  * Tong and Zhang (2023) Yaojun Tong and Lixin Zhang. 2023.  Discovering the next decade’s synthetic biology research trends with chatgpt.  _Synthetic and Systems Biotechnology_ , 8(2):220.
  * Wang et al. (2021) Benyou Wang, Qianqian Xie, Jiahuan Pei, Prayag Tiwari, Zhao Li, et al. 2021.  Pre-trained language models in biomedical domain: A systematic survey.  _arXiv preprint arXiv:2110.05006_.
  * Wang et al. (2022) Mingyang Wang, Zhe Wang, Huiyong Sun, Jike Wang, Chao Shen, Gaoqi Weng, Xin Chai, Honglin Li, Dongsheng Cao, and Tingjun Hou. 2022.  Deep learning approaches for de novo drug design: An overview.  _Current Opinion in Structural Biology_ , 72:135–144.
  * Wang et al. (2019) Sheng Wang, Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang. 2019.  Smiles-bert: large scale unsupervised pre-training for molecular property prediction.  In _Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informatics_ , pages 429–436.
  * Weininger (1988) David Weininger. 1988.  Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules.  _Journal of chemical information and computer sciences_ , 28(1):31–36.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint_.
  * Xia et al. (2017) Yingce Xia, Tao Qin, Wei Chen, Jiang Bian, Nenghai Yu, and Tie-Yan Liu. 2017.  Dual supervised learning.  In _Proceedings of the 34th ICML_ , pages 3789–3798.
  * Yuan et al. (2021) Zheng Yuan, Yijia Liu, Chuanqi Tan, Songfang Huang, and Fei Huang. 2021.  Improving biomedical pretrained language models with knowledge.  In _Proceedings of the 20th Workshop on Biomedical Language Processing_ , pages 180–190.
  * Zeng et al. (2022) Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2022.  A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals.  _Nature communications_ , 13(1):1–11.
  * Zhang et al. (2021) Xiao-Meng Zhang, Li Liang, Lin Liu, and Ming-Jing Tang. 2021.  Graph neural networks and their current applications in bioinformatics.  _Frontiers in genetics_ , 12.

##  Appendix A Pre-training Corpus

For the pre-training period, we obtain the text corpus from S2orc Lo et al.
(2020), which contains over 12.7 million academic papers. We download SMILES
strings from the PubChem database Kim et al. (2016), which contains billions
of substance records. The data between different tasks do not overlap.

For the SMILES corpus, we conduct three different tasks. (1) SMILES MLM: We
sample SMILES strings for 300k molecules to conduct the basic MLM task. (2)
Molecular property prediction: We collect 98k items of physical and chemical
properties from wet-lab experiments of 31k molecules from PubChem. There are
altogether 15 types of properties including Solubility, Color/Form, Boiling
Point, Flash Point, Density, Vapor Density, Decomposition, Corrosivity,
Melting Point, LogP, Vapor Pressure, Stability/Shelf Life, Odor, Taste, and
pH. Notice that we also remain the SMILES-name pairs of these molecules to
conduct mapping correlation learning. (3) Spatial structure learning: We
retain the other 300k molecules from PubChem. 15% of the atoms in each
molecule are randomly marked and required to recognize their connected atoms,
aromaticity and ring formation information.

For the text corpus, we conduct two types of tasks. (1) Text MLM: We select
400k literature abstracts in the chemistry and biomedicine field for the basic
MLM task. (2) Mapping correlation learning: There are two methods for
automatically obtaining parallel data between text and molecules. The first
one is to directly match the name-SMILES pairs, and we sample 200k molecules
for it. The second one is to roughly annotate the corresponding entity SMILES
in the corpus. Specifically, we adopt SciSpacy Neumann et al. (2019) to find
out the chemical entities, which are then matched with the list of substances
and synonyms from PubChem. In this way, we get over 2.4 million mentions of
4.7k most frequently-appeared molecules linked with their SMILES strings in
760k literature abstracts.

For the dual augmentation, we adopt the 14.7k molecules from ChEBI-20 train
set that are not contained in ChEBI-dia as the augmented pool, while do not
use their text descriptions to avoid introducing extra information.

##  Appendix B Intermediate turn analysis

We provide the intermediate prediction results of the multi-turn molecule
generation task in Table 5. Since the intermediate answers are annotated
automatically, their accuracy cannot be guaranteed and should be used as a
reference only. From the results we can conclude that: (1) We use the MolT5
initial checkpoint to annotate intermediate answers, and naturally the fine-
tuned MolT5 generates similar molecules and achieves a quite satisfying
performance. Though ChatMol gets a lower score, the plugin version ChatMol+
does even better than MolT5, showing the great assistance that the chemical-
natural language mapping tool (chemical entity recognition toolkits and SMILES
KBs) can provide; (2) Early turns have high accuracy, indicating that it is
easy for the models to find molecules that meet the requirements for the
general descriptions, while it is more difficult to complete the modification
for the supplementary fine structure descriptions.

Model | Turn | EM | BL | RDK
---|---|---|---|---
ChatMol+ | 1 | 0.402 | 0.716 | 0.672
2 | 0.211 | 0.691 | 0.592
3 | 0.190 | 0.716 | 0.614
4 | 0.125 | 0.716 | 0.626
ChatMol | 1 | 0.393 | 0.568 | 0.638
2 | 0.192 | 0.532 | 0.573
3 | 0.092 | 0.522 | 0.578
4 | 0.176 | 0.587 | 0.636
MolT5 | 1 | 0.384 | 0.689 | 0.635
2 | 0.194 | 0.650 | 0.559
3 | 0.135 | 0.657 | 0.536
4 | 0.206 | 0.717 | 0.598
T5 | 1 | 0.367 | 0.676 | 0.630
2 | 0.162 | 0.630 | 0.540
3 | 0.130 | 0.669 | 0.539
4 | 0.118 | 0.678 | 0.503
Table 5: Intermediate results for molecule generation.

##  Appendix C Training Setting Supplementary

We initialize ChatMol with the pre-trained T5-base (with the hidden size of
$768$). Other hyperparameter details can be found in the original paper Raffel
et al. (2020). Molecule understanding and generation correspond to two
different checkpoints. We implement our method in the PyTorch Paszke et al.
(2019) framework, and adopt the Huggingface Transformers Wolf et al. (2019).
We take the AdamW optimizer Loshchilov and Hutter (2019) which is suitable for
most of the PTMs in the T5 backbone.

For the multi-task pre-training, we add different task prefixes (e.g. “Predict
Solubility:”) to the input, and randomly mix up the training data for MLM,
SMILES-text parallel generation, property and spatial information prediction.
In this period, we set the learning rate as $5e-4$ and batch size as $256$.

For the downstream molecule understanding fine-tuning, we set the maximum
epoch number as $50$, and the early stop epoch number as $5$. The learning
rate and batch size is searched in $\left\\{1e-3,5e-4,2e-4\right\\}$ and
$\left\\{8,16,32\right\\}$, and eventually decided as $5e-4$ and $16$. Notice
that our hyper-parameter settings are different from the MolT5 paper Edwards
et al. (2022), therefore the results are higher than original reported. For
molecule generation, the learning rate and batch size is searched in
$\left\\{1e-3,5e-4,1e-4,5e-5\right\\}$ and $\left\\{16,32,64\right\\}$, and
eventually decided as $5e-4$ and $32$.

We train our models on a NVIDIA A100 SXM4 40 GB GPU. Evaluation metrics such
as BLEU and ROUGE are achieved with the assistance of NLTK toolkit Loper and
Bird (2002).

##  Appendix D Licenses and Data Usage Policy

PCdes, ChEBI-20 and S2orc are released under the CC BY-SA 4.0 license.
MoleculeNet are released under the MIT license. All the datasets are used in a
way consistent with their intended use. We observe the data samples and do not
find any offensive content or identifiers in these datasets.