# Enhancing Molecular Property Prediction via Mixture of Collaborative Experts

Xu Yao1, Shuang Liang1, Songqiao Han1,2,*, Hailiang Huang1,2,* * Corresponding
author. 1AI Lab, Shanghai University of Finance and Economics, Shanghai, China
2MoE Key Laboratory of Interdisciplinary Research of Computation and Economics
{yaoxu<EMAIL_ADDRESS>{han.songqiao<EMAIL_ADDRESS>
###### Abstract

Molecular Property Prediction (MPP) task involves predicting biochemical
properties based on molecular features, such as molecular graph structures,
contributing to the discovery of lead compounds in drug development. To
address data scarcity and imbalance in MPP, some studies have adopted Graph
Neural Networks (GNN) as an encoder to extract commonalities from molecular
graphs. However, these approaches often use a separate predictor for each
task, neglecting the shared characteristics among predictors corresponding to
different tasks. In response to this limitation, we introduce the GNN-MoCE
architecture. It employs the Mixture of Collaborative Experts (MoCE) as
predictors, exploiting task commonalities while confronting the homogeneity
issue in the expert pool and the decision dominance dilemma within the expert
group. To enhance expert diversity for collaboration among all experts, the
Expert-Specific Projection method is proposed to assign a unique projection
perspective to each expert. To balance decision-making influence for
collaboration within the expert group, the Expert-Specific Loss is presented
to integrate individual expert loss into the weighted decision loss of the
group for more equitable training. Benefiting from the enhancements of MoCE in
expert creation, dynamic expert group formation, and experts’ collaboration,
our model demonstrates superior performance over traditional methods on 24 MPP
datasets, especially in tasks with limited data or high imbalance.

###### Index Terms:

Molecular Property Prediction, Mixture of Experts, Graph Neural Networks

##  I Introduction

The discovery of lead compounds in drug development often incurs significant
time and financial costs [1]. Molecular Property Prediction (MPP) plays a
crucial role in this process, aiming to identify lead compounds with desired
properties within candidate molecules. These properties include factors such
as absorption, distribution, metabolism, and excretion (ADME), as emphasized
in [2]. Given the potential of deep learning methods in addressing complex
problems, numerous approaches based on deep learning have been proposed in
recent years for MPP tasks. Many of these methods leverage information, such
as molecular structure and elemental composition, to predict potential
properties, because the chemical properties of molecules are often closely
related to their structure [3]. As depicted in Fig. 1, molecular structures
can be naturally represented as graph-structured data, with atoms as nodes and
chemical bonds as edges. Hence, studies commonly treat MPP tasks as graph
prediction tasks, with our paper specifically emphasizing the graph
classification task.

Figure 1: Molecular Property Prediction in Drug Development.

Current graph-based MPP models can be conceptualized as the structure in Fig.
2(a), comprising two key components: Graph Neural Networks (GNNs) serving as a
universal molecular encoder, and standalone feedforward networks acting as a
task-specific predictor. Most of the works explores different GNN structures
to enhance the graph representation capability of encoder, such as Gated GNN
[4] or attention-based structures [5, 6, 7, 8, 9]. Due to the high cost of
biochemical experiments, MPP tasks face issues of data scarcity and imbalance,
manifesting as either insufficient overall data or a relative lack of target
class data. To overcome these challenges, some works [10] incorporates
additional information, like chemical knowledge graphs[11], into the GNN
encoder through pretraining strategies, as illustrated in Fig. 2(b). Some
other studies discuss the fusion for various tasks, utilizing a multi-tower
structure [12] as Fig. 2(c), or leveraging inputs with distinct task
identifiers [13] as Fig. 2(d). However, these methods primarily concentrate on
enhancing the common molecular traits by the GNN encoder, often neglecting the
exploration of shared elements in predictors corresponding to different tasks.

Figure 2: The Model Structures Related to Molecular Property Prediction.

In this paper, our objective is to enhance the performance of MPP by
leveraging the commonalities among molecules and their associations with
tasks. We propose the GNN-MoCE architecture based on GNN and Mixture of
Collaborative Experts (MoCE), which follows the structure in Fig. 2(f). In
this architecture, GNN serves as an encoder to discover shared traits among
molecular structures, while MoCE serves as a universal predictor for various
tasks, establishing the connection between molecular prediction and tasks. The
MoCE gives particular emphasis to the homogeneity issue in the expert pool and
the decision dominance dilemma within the expert group, which are often
overlooked by traditional MoE structures. On the one hand, MoCE considers the
absence of diversity that occurs when experts involved in decision-making
receive similar samples and subsequently tend to converge to similar states
after training. To handle this problem, we introduce Expert-Specific
Projection, leveraging Self-Attention Graph (SAG) pooling [14] and our
Attention Cosine Loss to provide each expert with a unique projection
perspective, thereby enhancing expert diversity and allowing experts to unlock
collaborative potential within the entire expert pool. On the other hand, MoCE
also considers the decision dominance within the dynamic decision group. This
phenomenon manifests as a specific expert attaining a dominant weight
suppresses the learning of other experts in the decision group, leading to the
degradation of a decision group into one expert. To alleviate the impact of
decision dominance, we propose the Expert-Specific Loss, which emphasizes more
equitable training for each expert within the decision group, thereby
promoting collaborative effectiveness.

The primary contributions of this study can be summarized as follows: (1) We
present the GNN-MoCE architecture, an efficient approach for molecular graph
classification that leverages both GNN and MoCE, and is further enhanced by
related tasks. GNN captures common molecular patterns, while our implemented
MoCE structure establishes connections between molecular structures and tasks.
(2) During the expert generation, we boost diversity by incorporating the
Expert-Specific Projection. The expert decision group is then formed by
leveraging task description embedding, which semantically enhances the
importance and load loss. To encourage collaboration within the group, we
apply our Expert-Specific Loss, ensuring a equitable training for the experts
involved in decision-making. (3) We conducted experiments on 24 datasets,
demonstrating the superior performance of our approach, particularly on
datasets characterized by limited samples or extreme class imbalance. We have
open-sourced the proposed method and all testing datasets, available at:
https://github.com/Hyacinth-YX/mixture-of-collaborative-experts.

##  II Related Work

MPP involves the task of forecasting the physical and chemical properties of
molecules based on their internal information. Previous research in the field
can be categorized into two major classes: the classical method utilizing non-
graph features such as quantitative values or sequential information, and the
graph-based method employing graph features.

###  II-A Classical Methods in MPP

Prior to the surge of deep learning, researchers would rely on physical
knowledge and employ Density Functional Theory (DFT) calculations, as
demonstrated by [15], to compute molecular properties. Alternatively, methods
such as Quantitative Structure-Activity Relationship (QSAR) [16] utilize
theoretical calculations and statistical analysis tools to construct
mathematical models that capture the relationship between molecular structure
and properties. However, the computational costs associated with DFT and QSAR
are both notably high [17]. With the advancement of deep learning, more and
more efforts have incorporated deep learning methods to aid in predicting
molecular properties [18]. Some works conducted prediction of molecular
properties using sequence modeling, given that molecules could be effectively
represented as sequences, such as SMILES expressions [19] or molecular
fingerprints [20]. Recent works, exemplified as [21, 22, 23], have
demonstrated the utilization of sequence modeling in predicting molecular
properties. Specifically, [21] employed the BERT [24] model, while [22, 23]
utilized CNN [25] and Feed-Forward Network for sequence modeling,
respectively.

###  II-B Graph-based Methods in MPP

Moleculars can be naturally encoded as graphs, with atoms serving as nodes and
bonds forming edges. Since GNN have demonstrated robust representation
capabilities for graph-structured data, GNN-based MPP methods have gradually
emerged as the mainstream in MPP research. As illustrated in Fig. 2(a), we
decompose the model architecture of GNN-based MPP methods into two main
components: an encoder and a predictor. The predictor is often a relatively
simple linear layer used for result prediction. In contrast, the encoder is
the primary focus of existing research, which is commonly viewed as a Message
Passing Graph Neural Network (MPNN) [26] or its variants. Current research
primarily revolves around enhancing the representational capabilities of GNN
structures. For instance, [6] proposed a model called Chemprop based on
Directed MPNN (D-MPNN) [27] to predict molecular properties. Considering the
characteristics of MPP tasks, Chemprop incorporates 200 globally calculable
molecular features to enhance the performance of MPP tasks. As the attention
mechanism [28] becomes increasingly prevalent in deep learning, many MPP
studies incorporate attention mechanism into GNN models, as seen in works such
as [29, 30, 31, 32]. In [30], besides attention mechanism, the edge memory
schemes are also introduced to further strengthen the representation
capability of chemical bonds in molecular graphs. [31] employs attention
mechanism to interpretably learn the relationship between chemical properties
and structures. Additionally, [32] introduces a Gated GNN [4] structure to
enhance graph representation capabilities.

In overcoming the data scarcity and imbalance in MPP, certain endeavors aim to
enhance GNN encoder by incorporating additional information. Broadly, these
efforts can be categorized into two major classes: pretraining on external
information and joint training for diverse internal tasks.

####  II-B1 Enhanced via External Pretraining

As illustrated in Fig. 2(b), some research focuses on augmenting molecular
embeddings through pretraining graph encoders with novel knowledge. Some
approaches introduce new views during pretraining. This strategy allows for
more effective leveraging of intrinsic molecular characteristics. [33] employs
a combination of Gated Recurrent Unit (GRU) [34] and Transformer [28]
structures to capture both local and global information in molecules,
facilitating the combination of molecular information at both scales. [7]
integrates SMILES and molecular graph representations, achieving an
integration of sequence information and molecular graph information through
unsupervised reconstruction. [9] introduces 3D molecular information into the
model, endowing the model with the capability to infer 3D molecular structures
from 2D information. Other works related to pretraining focus on incorporating
knowledge from diverse domains. For instance, [35] leverages relationships
between molecular properties revealed by prior studies in physical chemistry
for learning. [8] utilizes knowledge derived from chemical reaction
expressions, employing chemical reactions for pretraining.

####  II-B2 Enhanced through Internal Tasks

Existing research [36] indicates that current unsupervised pre-training
methods offer limited improvement for downstream tasks, falling short of the
gains achieved by supervised pre-training. Supervised learning across various
tasks may serve as a supplement to further enhance the effectiveness of MPP. A
frequently employed architecture is the multi-tower model with parameter
sharing, depicted in Fig. 2(c). This structure is utilized by [12] in their
work. The shared encoder processes input molecules and encodes them into a
unified representation across various tasks. The model concurrently performs
multiple tasks, providing diverse outputs, from which users can select the
relevant portions. Another variant structure is illustrated in Fig. 2(d), as
employed by [13] in their work with the task encoding. This approach employs
one-hot encoding to map different tasks into a vector space, acquiring task
embeddings through a GNN. By incorporating a task identifier in the input, the
model can generate a unified output representing results for different tasks,
thereby gaining the capability to leverage inter-task relationships.

In summary, the existing research has primarily focused on enhancing the
expressive capabilities of GNN for molecular representation, aiming to improve
the effectiveness of MPP tasks. However, there is still an insufficient
exploration into how models from diverse tasks can synergize to enhance
ensemble effects. Based on MoE, our approach can be viewed as a unified model
for integrating sub-models trained on diverse tasks, providing a basis for
discussing the collaboration among sub-models.

###  II-C MoE Structure

MoE [37] has demonstrated distinct advantages in leveraging task distinctions
to extract specific task nuances. An example is the Multi-gate Mixture-of-
Experts (MMoE) [38], which is depicted in Fig. 2(e). In contrast to the
classical parameter sharing models, MMoE employs multiple gate layers to route
features to different tasks. As a result, MMoE can effectively discriminate
between different tasks. Unlike the use of MoE in the encoder as MMoE, we
retain the GNN structure in our encoder, and replace the task-specific
predictors with a shared expert pool. This allows us to facilitate task
collaboration using simple Multi-Layer Perceptron (MLP) experts.

Figure 3: Architecture of GNN-MoCE

##  III Architecture

###  III-A Problem Statement

The objective of MPP is to predict molecular properties, such as toxicity or
activity. A dataset with a size of $N$ can be represented as
$\\{(x_{i},y_{i},t_{i})\\}_{i=1}^{N}$, comprising molecular graph features
$x_{i}$, corresponding labels $y_{i}$, and task description embeddings
$t_{i}$. Molecular graph features $x_{i}$ encompass node and edge attributes,
such as atomic types and bond types.

###  III-B Network Architecture

Expanding on the traditional message-passing GNN structure, we introduce an
architecture named GNN-MoCE. Diverging from other studies that primarily focus
on improving graph embeddings, our emphasis lies in the collaboration within
the MoCE. Here, GNNs are solely responsible for embedding graph information,
while the experts in the MoCE layer are assigned the task of predicting task-
specific outcomes. Fig. 3 illustrates the overall architecture of GNN-MoCE.

####  III-B1 Inputs Embedding

The model inputs consist of two components: molecular graph information and
task information. The molecular graph information encompasses common features
of atoms and chemical bonds, while the task information is derived from the
natural language descriptions of tasks in the dataset. By embedding the
natural language descriptions of tasks, we can obtain the semantic
representation of tasks, which plays a crucial role in forming the dynamic
expert decision group. This is because experts who perform similar tasks are
more likely to achieve better performance through collaboration, and task
semantics provide a good perspective for measuring the degree of task
similarity. These features are fed into the stacked processing layers of GNN-
MoCE. Each layer undertakes the tasks of graph embedding and result
prediction, which correspond to the GNN Encoder and MoCE Predictor within the
component, respectively.

####  III-B2 GNN Encoder

GNN Encoder consists of multiple GNN modules based on the message-passing
structure. In our work, we configure the GNN module as Graph Isomorphism
Network (GIN) layers [39]. During the message propagation phase, the state of
graph nodes is iteratively updated based on the states of their neighboring
nodes and the features of connecting edges. For a given node $v$, its hidden
state $p^{(t+1)}$ at time $t+1$ is defined as follows:

$m^{(t+1)}_{v}=\sum_{w\in N(v)}M_{t}(p^{t}_{v},p^{t}_{w},e_{vw}),$ (1)
$p^{(t+1)}_{v}=U(p^{t}_{v},m^{(t+1)}_{v}).$ (2)

Where $N(v)$ represents all neighboring nodes of $v$, and $m_{v}^{(t+1)}$ is
the $t+1$ layer message that aggregates embeddings of all neighboring nodes
and edge embeddings. $p_{i}^{t}$ and $e_{ij}$ represent the embeddings of node
$i$ and edge $ij$, respectively. Using the update function $U$, we obtain
embeddings for the $t+1$ layer. Theoretically, the GNN Module can be replaced
with other pre-trained GNN embedding modules proposed in the existing
literature.

####  III-B3 MoCE Predictor

The MoCE Predictor, designed based on the MoE structure, makes decisions for
samples by the dynamic decision group. Each prediction involves assigning
appropriate experts from the expert pool to form the expert group. We will
introduce the predictor in the following three parts: the input of the expert
group, the formation of the expert group, and the weighted voting of the
expert group.

The inputs of the expert group include the node embeddings of the graph and
the task description embeddings. The node-level embeddings of the $i$-th graph
need to be read out as graph-level information $\hat{x}$ through a graph
readout module $R$ first.

$\hat{x}_{i}=R(x_{i}).$ (3)

In the MoCE architecture, we employ two distinct graph readout modules. The
first one operates before the sample is routed to an expert, requiring the
embedding of nodes through Global Mean Pooling (GMP) to obtain a graph-level
embedding. This enables the routing module to allocate the sample to an
appropriate expert based on the overall characteristics of the graph. The
second readout module, referred to as SAG pooling, is associated with each
expert. In this context, some unique parameters are employed for each expert,
thereby guaranteeing that each expert possesses an individualized graph-
reading mechanism. Details of the latter are elaborated in Section IV-A.

The formation of the expert group can also be viewed as the process of routing
the sample to the top-$k_{s}$ experts. This process is related to both the
graph information and task information of the sample. The $i$-th sample
consists of two parts: the graph-level readout embedding
$x_{i}\in\mathbb{R}^{e_{f}}$ and the task description embedding
$t_{i}\in\mathbb{R}^{e_{t}}$. Here, $e_{f}$ and $e_{t}$ represent the
dimensions of graph features and task embeddings, respectively. Upon receiving
the current layer’s graph and task embedding, the MoCE’s routing module
directs samples to the appropriate experts. Gating scores
$H(\hat{x}_{i},t_{i})\in\mathbb{R}^{m}$ for pre-defined $m$ experts are
computed using graph-level embedding $\hat{x}_{i}$ and task embedding $t_{i}$.
We denote $h_{ij}$ as the score of sample $s_{i}$ for the $j$-th expert.
Following the noise mechanism described in [37], we assume
$h_{ij}\sim\mathcal{N}(\mu_{ij},\sigma_{ij})$, where $\mu_{ij}$ and
$\sigma_{ij}$ are estimated as follows:

$\mu_{ij}=(\Gamma(W_{\mu_{1}}^{T}\hat{x}_{i})+W_{\mu_{2}}^{T}t_{i})_{j},$ (4)
$\sigma_{ij}=\text{Softplus}(\Gamma(W_{\sigma_{1}}^{T}\hat{x}_{i})+W_{\sigma_{2}}^{T}t_{i})_{j}.$
(5)

Where $W_{\mu_{1}},W_{\sigma_{1}}\in\mathbb{R}^{e_{f}\times m}$ are learnable
parameters for mapping sample feature embeddings, and
$W_{\mu_{2}},W_{\sigma_{2}}\in\mathbb{R}^{e_{t}\times m}$ are learnable
parameters for mapping task embeddings. $\Gamma(\cdot)$ is a specific mapping
function that retains the top $k_{t}$ values in a vector and fills the rest
with the minimum value. This function is used to constrain expert selection to
be within the top $k_{t}$ experts preferred by the task, while still allowing
some flexibility for routing to experts preferred by other samples.

Therefore, $h_{ij}$ can be expressed as follows, where $StdNormal()$
represents random value generated from a standard normal distribution.

$h_{ij}=\mu_{ij}+\sigma_{ij}\cdot\text{StdNormal}().$ (6)

After obtaining $h_{i*}=[h_{i1},h_{i2},\dots,h_{im}]$, we retain the top
$k_{s}$ values and set the rest to zero, resulting in the final gate
$G(x_{i},t_{i})$ of sample $i$:

$G(x_{i},t_{i})=\text{Softmax}(\text{KeepTopK}(h_{i*})).$ (7)

The aforementioned routing mechanism is visually depicted in Fig. 4. Both the
graph embedding and task embedding undergo a routing gate to be mapped into
expert scores. For task routing scores, with the existence of the minimum
mask, experts outside of the top-$k_{t}$ are filled with the minimum expert
score. After adding the task routing scores to the sample routing scores, the
final routing score matrix is inclined towards the top-$k_{t}$ experts. This
achieves the adjustment of expert selection based on the task embedding. Based
on the final routing score matrix, the top-$k_{s}$ experts with high routing
scores will be assigned to this sample.

Figure 4: The process of routing samples with task information to experts.

The weighted voting of the expert group is implemented based on the gates
calculated above, and the expert’s corresponding gate is essentially its
weight in the voting process. Let $g_{j}(\hat{x}_{i},t_{i})\in
G(\hat{x}_{i},t_{i})$ represent the gate scalar for sample $i$, and
$E_{j}(\tilde{x})$ denote the output of the $j$-th expert for this sample. We
can obtain the integrated result from $m$ experts in this MoCE layer as:

$y_{i}=\sum_{j}^{m}g_{j}(\hat{x}_{i},t_{i})E_{j}(\tilde{x}_{ij}).$ (8)

It is noteworthy that each expert is assigned distinct learnable parameters of
SAG readout module. Hence, the graph input for expert $j$ in (8) is actually
the graph representation $\tilde{x}_{ij}=R_{j}(x_{i})$ obtained after applying
its own readout.

####  III-B4 Outputs Integration

Each component predicts the results based on the molecular embeddings obtained
at that layer, and these predictions need to be collected and integrated at
the end. As model’s preferences for information at each layer may vary across
tasks, the aggregation of these results needs to be adjusted based on the task
embedding information.

For the $i$-th sample, its corresponding task embedding is denoted as
$t_{i}(i\in 1,\dots,n)$. Let $Map(*)$ represent the mapping function that
assigns weights to results based on the task embedding. We define $W_{i}$ as
the mapping weight of $i$-th sample.

$W_{i}=\text{Softmax}(\text{Map}(t_{i})).$ (9)

Consequently, the final output is given by,

$r_{i}=\sum_{l}^{m}(w_{il}o_{il}).$ (10)

In this formula, $w_{il}\in W_{i}$ corresponds to the weight of the $l$-th
layer. $o_{il}$ denotes the result of the $l$-th MoCE layer for the $i$-th
sample, where $l$ ranges from 1 to $L$, representing the total layers number.

##  IV Collaborative Experts

The traditional Sparsely Gated MoE structure lacks consideration for
collaboration among experts. This deficiency can be viewed from both global
and local perspectives. Globally, experts share the same input, and the
distinctions between experts primarily rely on the random initialization of
parameters. This results in experts having very similar parameters, leading to
unstable performance and difficulty in achieving ensemble benefits. Locally,
we define the dynamic combination of the top-$k_{s}$ experts that collectively
vote on a particular sample as a dynamic expert group. We observe that
traditional MoE structures often exhibit a dominant expert with extremely high
weights in the expert group. This dominance suppresses the learning of other
experts in the group, making the decision group degrade to one single expert
and diminishing the effectiveness of collaboration. To address these issues
and enhance expert collaboration on both global and expert group levels, we
propose two methods: Expert-Specific Projection and Expert-Specific Loss. Fig.
5 depicts these two methods.

Figure 5: Methods for Collaborative Experts.

###  IV-A Global Collaboration

The classical MoE structure exhibits a limitation in providing diverse inputs
to individual experts, potentially leading to limited diversity among experts.
This consequently impacts the global collaboration effectiveness of experts.
To address this issue, we propose the method of Expert-Specific Projection,
assigning each expert its unique perspective on viewing samples. In Fig. 6, we
intuitively illustrate the significance of expert diversity and explain how
our Expert-Specific Projection contributes to enrich expert diversity. Fig.
6(a) depicted a simple binary classification scenario. In a conventional MoE
structure, the group of experts make decisions together may be allocated to
similar sample spaces, leading to a learned state illustrated in Fig. 6(b). It
is evident that the learning of experts is constrained by the diversity of
samples, making it challenging to achieve improved ensemble performance
through voting. Contrastingly, by assigning distinct projection planes to each
expert, enabling them to learn classifiers on these planes, we obtain diverse
classification planes that each expert may fit, as illustrated in Fig.
6(c1-c3). The integration of these experts is depicted in Fig. 6(c),
demonstrating superior weighted classification performance compared to
traditional MoE. Therefore, by making the projection planes distinct, we can
enrich expert diversity as much as possible, fostering collaboration among
experts. Our Expert-Specific Projection method is designed following this
rationale.

Figure 6: Projection of Experts.

In detail, we utilize the SAG pooling approach [14] to project the generic
node representations obtained from the graph encoder onto individual
perspectives of each expert, thereby obtaining distinctive graph-level
representations. In contrast to traditional global graph readout methods like
Global Mean Pooling (GMP) [40], SAG pooling has the capability to selectively
read graph information based on a learnable vector
$\Theta_{att}\in\mathbb{R}^{F\times 1}$. Specifically, the self-attention
scores $\tilde{Z}\in\mathbb{R}^{N\times 1}$ for each node on the graph are
computed as follows:

$Z=X\Theta_{att}$ (11) $\tilde{A}=A+I_{N}$ (12)
$\tilde{Z}=\sigma(\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}Z)$
(13)

In (11), $X\in\mathbb{R}^{N\times F}$ represents the feature embedding matrix
for nodes, where $N$ is the number of nodes and $F$ is the feature dimension.
Equation (11) is the main component for calculating attention scores,
resulting in $Z\in\mathbb{R}^{N\times 1}$ as the basic attention scores for
each node under the given $\Theta_{att}$. However, due to the graph’s unique
structure, we need to adjust the attention scores to incorporate more
topological information using the adjacency matrix $A\in\mathbb{R}^{N\times
N}$. The adjacency matrix with self-connections $\tilde{A}$ is obtained as
(12). Additionally, $\tilde{D}$ represents the degree matrix corresponding to
$\tilde{A}$, and $\sigma$ denotes the activation function. Equation (13)
produces the attention scores adjusted for the graph structure.

Subsequently, a pooling ratio $\kappa\in(0,1]$ is defined to adjust the number
of subgraph nodes to be selected. Based on the obtained attention scores
$\tilde{Z}$, the top $\lceil\kappa N\rceil$ nodes are selected, and their
feature embeddings are aggregated, e.g., by computing their mean. The
aggregated result serves as the representation of the graph under a given
$\Theta_{att}$.

The reason for employing the SAG pooling method is that it enables the control
of the expert’s perspective in observing the samples by manipulating the value
of $\Theta_{att}$, thereby enhancing the capture of local details in the
graph. On the one hand, for information comprehensiveness, they can
collectively obtain information about the entire graph when there are a
sufficient number of experts. On the other hand, for information diversity,
differentiating the $\Theta_{att}$ values can make the inputs to the experts
more diverse. A variety of expert input views can further improve the overall
performance of the system.

To ensure that the perspectives of the experts are as distinct as possible, we
add an Attention Cosine Loss term $\mathcal{L}_{att}$ to the overall loss. For
M experts and an embedding dimension of F, we normalize $\Theta_{att_{i}}$ as
follows:

$\tilde{\Theta}_{att_{i}}=\frac{\Theta_{att_{i}}}{||\Theta_{att_{i}}||_{2}}$
(14)

Then, we define the Attention Cosine Loss term as:

$\mathcal{L}_{att}=\frac{1}{M^{2}}\sum_{i}^{M}\sum_{j}^{M}\tilde{\Theta}_{att_{i}}^{T}\tilde{\Theta}_{att_{j}}.$
(15)

Here, $\frac{1}{M^{2}}$ is a scaling factor to stabilize the numerical values
of the loss term.

###  IV-B Group Collaboration

In our research, we observed the presence of a dominant expert with extremely
high weights in the expert group of the classical MoE structure. This dominant
expert hinders the training of other experts within the same group, thereby
affecting the overall decision-making effectiveness of the expert group. To
address this issue, we introduce the Expert-Specific Loss, ensuring that each
expert within the group undergoes fair training. In Fig. 7, we illustrate the
voting weights of experts in the decision group in traditional MoE and MoE
with our proposed loss, respectively. It is evident that in the original MoE
structure, one expert typically occupies close to 80% of the weight, with only
20% of the weight distributed among other experts. However, upon incorporating
our Expert-Specific Loss, each expert receives equitable training, leading to
an increase in the weights of other experts. This results in a more
significant contribution to the decision-making process and prevents the
degradation of the expert group into a single expert.

Figure 7: Voting Weights of Experts in Decision Group

To be more specific, rather than solely optimizing the final weighted output
like other works [41, 37, 42], we integrate the individual loss of each expert
into the loss for ensemble weighted outputs. In other words, this entails
enhancing the individual task performance of each expert. Formally, we augment
the ultimate optimization loss by introducing the Expert-Specific Loss
$\mathcal{L}_{exp}$. This term is defined as follows:

$\mathcal{L}_{exp}=\sum_{i}^{M}\sum_{j}^{N}\mathbb{I}(i,j)BCE(\hat{y}_{ij},y_{j}).$
(16)

Where $N$ and $M$ represent the number of records and the number of experts,
respectively, and the indicator function $\mathbb{I}(i,j)$ indicates which
data point is assigned to which expert.

###  IV-C Overall Losses

The enhancement of expert collaboration in our model is primarily achieved by
incorporating auxiliary losses. In summary, our Collaborative Loss comprises
the following components:

$\mathcal{L}_{col}=\mathcal{L}_{att}+\mathcal{L}_{exp}+\mathcal{L}_{imp}+\mathcal{L}_{lod}.$
(17)

Subsequently, the Collaborative Loss is multiplied by a scalar $\beta\in(0,1]$
and then added to the model’s original base loss $\mathcal{L}_{base}$. This
operation yields our ultimate loss term, denoted as $\mathcal{L}_{overall}$:

$\mathcal{L}_{overall}=\mathcal{L}_{base}+\beta\cdot\mathcal{L}_{col}.$ (18)

In addition to the aforementioned $\mathcal{L}_{att}$ and $\mathcal{L}_{exp}$,
we introduce two supplementary loss $\mathcal{L}_{imp}$ and
$\mathcal{L}_{lod}$ to balance the samples and tasks distributed to experts.
The introduction of these two additional auxiliary losses is motivated by the
imbalance in the global allocation of samples and tasks, where a minority of
experts may be assigned a disproportionate amount of samples. This phenomenon
has been described in [43] and [44]. To address this imbalance issue, [37]
proposed two types of losses, namely Importance and Load losses. Building upon
this method, we incorporate task information into these two losses, ensuring a
more balanced allocation of both tasks and samples. Although they are not the
core components of our study, they play a complementary role in improving the
model’s effectiveness.

####  IV-C1 Task-adjusted Importance Loss

With the computed gate $G(x,t)$ in section III-B3, we determine the importance
of each expert as the sum of its gate’s score in a batch. Then, we balance the
distribution of expert selection by minimizing the coefficient of variation of
expert importance, which is specifically defined as the Importance loss
$\mathcal{L}_{imp}$:

$I_{j}=\sum_{i}G(x_{i},t_{i})_{j}$ (19)
$\mathcal{L}_{\text{imp}}=\beta\cdot\text{CV}([I_{1},I_{2},\dots,I_{m}])^{2}$
(20)

Where $\beta$ serves as a scaling factor for the loss, and CV represents the
coefficient of variation. This loss term ensures a more balanced distribution
of expert weights within the same batch.

####  IV-C2 Task-adjusted Load Loss

The load balance loss term $\mathcal{L}_{lod}$ is used to balance the number
of samples each expert receives. This is achieved by balancing the
probabilities of experts selecting samples. Since a sample $i$ is chosen by
expert $j$ only when $h_{ij}$ is the $k$-th greatest element in $h_{i\cdot}$,
it implies that when $h_{ij}$ exceeds a threshold $h_{k_{ij}}$ (representing
the k-th largest value in $h_{i\cdot}$ but excluding $h_{ij}$), the
probability of sample $i$ being chosen by expert $j$ can be defined as:

$P(x_{i},t_{i})_{j}=Pr(h_{ij}>h_{k_{ij}}).$ (21)

As $h_{ij}$ is modeled in our system to follow
$\mathcal{N}(\mu_{ij},\sigma_{ij})$, this probability can be expressed as:

$P(x_{i},t_{i})_{j}=\Phi(\frac{\mu_{ij}-h_{k_{ij}}}{\sigma_{ij}}).$ (22)

Similar to the definition of the Importance loss, we use the coefficient of
variation to define the final Load loss:

$L_{j}=\sum_{i}P(x_{i},t_{i})_{j}$ (23) $\mathcal{L}_{lod}=\beta\cdot
CV([L_{1},L_{2},\dots,L_{m}])$ (24)

In summary, through the optimization of importance and load loss with task
parameters, we facilitate sample routing based on task information. This
approach enables a broader participation of experts in tasks, thus reducing
the risk of these experts degenerating into a tiny subset.

##  V Experiments

###  V-A Experimental Setting

####  V-A1 Datasets

We conducted experiments on 24 MPP datasets sourced from Therapeutics Data
Commons [2]. These encompass diverse tasks such as ADME (absorption,
distribution, metabolism, and excretion) properties, Tox Prediction, and High-
throughput Screening Prediction. Given that Tox21 is a multi-label dataset, we
partitioned the dataset into 12 subtasks, resulting in training and validation
across 35 tasks. In pursuit of conciseness, for the Tox21 dataset, we only
report the average effectiveness across different tasks of Tox21. We conducted
separate training for baseline methods, while jointly trained on all 35 tasks
for our GNN-MoCE model.

For each dataset, we all applied stratified scaffold split [6] to get the
develop (80%) and test (80%) datasets. Within the develop Datasets, the train
and validation datasets respectively constitute 80% and 20% of the total. The
scaffold split is a method of dataset partitioning based on molecular
scaffolds. In comparison to a purely random split, tasks generated through
scaffold split are more challenging, demanding higher model generalization
capabilities, and providing a better approximation to real-world data
scenarios. The stratified scaffold split we employ takes into account datasets
with limited quantities or significant imbalances. It performs scaffold splits
within the same category to preserve label distribution similarity to the
original dataset as much as possible. Additionally, to ensure the reliability
of experimental conclusions, our dataset partitioning is not fixed and varies
with different random seeds. In our experiments, we ensure that data
partitions remain consistent under the same random seed.

Figure 8: Similarity of Task Description Embeddings

####  V-A2 Feature Preparation

The model’s input encompass information about the molecular graph and the task
identifier.

For molecular graph information, we need to extract graph from sequence
representation, as the majority of datasets represent molecules in the form of
SMILES expressions [19, 2]. In this paper, we adopt the standard method
proposed by the Open Graph Benchmark (OGB) [45] to extract graphs from the
SMILES and embed molecules into a specified dimension.

For task identifier, the embedding process of task is relatively more complex.
If we simply enumerate all tasks and generate the task embeddings using the
one-hot method, like method in [13], it becomes challenging to capture the
semantic distinctions and relationships among tasks. In our model, we employ
natural language embeddings of task descriptions to guide expert routing. This
assists the model in selecting more semantic similar experts. Furthermore, as
natural language is human-readable, our model can be easily extended to other
datasets. In this experiment, we utilized the text-embedding-ada-002 model by
OpenAI [46] to embed task descriptions. The similarity of embeddings across
our 24 datasets is depicted in Fig. 8.

TABLE I: Detailed Results of 35 tasks Training Experiments (AUCROC $\uparrow$) Method | Dataset | HIV | SARSCoV2_VT∗ | Tox21 | ClinTox | SARSCoV2_3CD∗ | HIA_Hou | PAMPA∗ | CYP2D6_Veith
---|---|---|---|---|---|---|---|---|---
Ratio(Total) | 0.034(26320) | 0.057(949) | 0.064(4180) | 0.077(945) | 0.085(563) | 0.149(369) | 0.152(1301) | 0.193(8403)
Vanilla | CNN | 0.72±0.034 | 0.525±0.07 | 0.669±0.06 | 0.822±0.053 | 0.645±0.099 | 0.725±0.125 | 0.621±0.061 | 0.795±0.012
GIN | 0.704±0.045 | 0.554±0.041 | 0.728±0.054 | 0.816±0.055 | 0.719±0.081 | 0.93±0.046 | 0.601±0.042 | 0.81±0.026
GCN | 0.723±0.08 | 0.501±0.082 | 0.711±0.052 | 0.842±0.068 | 0.721±0.09 | 0.879±0.051 | 0.615±0.034 | 0.83±0.049
NeuralFP | 0.751±0.033 | 0.536±0.091 | 0.723±0.051 | 0.819±0.051 | 0.709±0.105 | 0.909±0.047 | 0.631±0.054 | 0.826±0.021
Chemprop | 0.748±0.026 | 0.563±0.071 | 0.754±0.047 | 0.836±0.027 | 0.715±0.083 | 0.829±0.049 | 0.659±0.067 | 0.856±0.013
AttentiveFP | 0.72±0.052 | 0.534±0.092 | 0.735±0.056 | 0.605±0.121 | 0.732±0.104 | 0.891±0.046 | 0.667±0.046 | 0.839±0.017
Pretrain | KCL | 0.75±0.029 | 0.577±0.083 | 0.715±0.043 | 0.594±0.069 | 0.713±0.099 | 0.887±0.044 | 0.657±0.035 | 0.801±0.022
ContextPred | 0.763±0.034 | 0.581±0.057 | 0.78±0.045 | 0.71±0.075 | 0.72±0.107 | 0.932±0.031 | 0.684±0.048 | 0.902±0.012
AttrMasking | 0.768±0.041 | 0.599±0.057 | 0.772±0.041 | 0.684±0.076 | 0.729±0.127 | 0.933±0.025 | 0.668±0.047 | 0.897±0.008
| GNN-MoCE(Ours) | 0.779±0.036 |  0.645±0.091 | 0.808±0.029 | 0.807±0.08 | 0.765±0.056 | 0.953±0.034 | 0.701±0.048 | 0.855±0.017
Method | Dataset | CYP2C9_SC∗ | Carcinogens∗ | Bio_Ma∗ | BBB_Martins | CYP2D6_SC∗ | hERG | SkinReaction | CYP2C9_Veith
Ratio(Total) | 0.203(428) | 0.229(179) | 0.247(409) | 0.248(1299) | 0.293(426) | 0.305(419) | 0.32(231) | 0.336(7738)
Vanilla | CNN | 0.617±0.042 | 0.832±0.099 | 0.653±0.075 | 0.835±0.047 | 0.659±0.054 | 0.695±0.056 | 0.146±0.157 | 0.849±0.014
GIN | 0.544±0.072 | 0.797±0.099 | 0.598±0.07 | 0.818±0.051 | 0.669±0.066 | 0.705±0.054 | 0.177±0.142 | 0.821±0.035
GCN | 0.589±0.015 | 0.772±0.043 | 0.588±0.044 | 0.852±0.105 | 0.684±0.049 | 0.711±0.072 | 0.243±0.051 | 0.861±0.143
NeuralFP | 0.565±0.036 | 0.838±0.081 | 0.607±0.063 | 0.838±0.053 | 0.701±0.052 | 0.718±0.068 | 0.193±0.12 | 0.862±0.016
Chemprop | 0.592±0.073 | 0.837±0.091 | 0.659±0.066 | 0.857±0.044 | 0.732±0.056 | 0.725±0.051 | 0.109±0.139 | 0.882±0.012
AttentiveFP | 0.586±0.064 | 0.853±0.072 | 0.655±0.089 | 0.824±0.054 | 0.726±0.055 | 0.796±0.066 | 0.392±0.289 | 0.866±0.015
Pretrain | KCL | 0.614±0.074 | 0.821±0.066 | 0.645±0.076 | 0.833±0.069 | 0.725±0.056 | 0.78±0.055 | 0.52±0.146 | 0.852±0.017
ContextPred | 0.547±0.075 | 0.834±0.073 | 0.62±0.094 | 0.871±0.043 | 0.731±0.064 | 0.764±0.062 | 0.347±0.185 | 0.927±0.007
AttrMasking | 0.608±0.025 | 0.857±0.071 | 0.625±0.101 | 0.849±0.037 | 0.724±0.056 | 0.792±0.045 | 0.346±0.146 | 0.921±0.011
| GNN-MoCE(Ours) | 0.644±0.065 | 0.935±0.028 | 0.731±0.068 | 0.902±0.047 | 0.819±0.042 | 0.835±0.049 | 0.51±0.076 | 0.902±0.011
Method | Dataset | CYP3A4_Veith | AMES | CYP2C19_Veith | CYP1A2_Veith | CYP3A4_SC∗ | Pgp_Bro∗ | hERG_Karim | DILI
Ratio(Total) | 0.408(7889) | 0.44(4553) | 0.447(8105) | 0.468(8050) | 0.479(428) | 0.489(779) | 0.492(8604) | 0.497(304)
Vanilla | CNN | 0.844±0.021 | 0.739±0.034 | 0.846±0.014 | 0.887±0.008 | 0.571±0.088 | 0.892±0.027 | 0.772±0.015 | 0.791±0.057
GIN | 0.833±0.023 | 0.752±0.024 | 0.832±0.033 | 0.865±0.027 | 0.526±0.078 | 0.849±0.042 | 0.761±0.024 | 0.77±0.079
GCN | 0.862±0.02 | 0.79±0.015 | 0.868±0.024 | 0.9±0.008 | 0.585±0.012 | 0.893±0.083 | 0.8±0.021 | 0.8±0.015
NeuralFP | 0.867±0.019 | 0.796±0.026 | 0.868±0.017 | 0.894±0.014 | 0.548±0.053 | 0.843±0.041 | 0.798±0.016 | 0.823±0.041
Chemprop | 0.883±0.018 | 0.821±0.017 | 0.884±0.009 | 0.908±0.012 | 0.529±0.068 | 0.759±0.043 | 0.822±0.018 | 0.824±0.065
AttentiveFP | 0.868±0.022 | 0.803±0.037 | 0.871±0.016 | 0.906±0.013 | 0.607±0.08 | 0.916±0.034 | 0.792±0.017 | 0.828±0.059
Pretrain | KCL | 0.841±0.026 | 0.745±0.03 | 0.843±0.011 | 0.873±0.017 | 0.596±0.077 | 0.92±0.041 | 0.786±0.021 | 0.791±0.049
ContextPred | 0.918±0.012 | 0.799±0.02 | 0.923±0.008 | 0.939±0.008 | 0.564±0.077 | 0.893±0.034 | 0.816±0.019 | 0.829±0.042
AttrMasking | 0.919±0.012 | 0.806±0.022 | 0.92±0.008 | 0.936±0.008 | 0.552±0.081 | 0.916±0.018 | 0.819±0.012 | 0.839±0.023
| GNN-MoCE(Ours) | 0.898±0.013 | 0.834±0.026 | 0.886±0.012 | 0.912±0.009 | 0.737±0.059 | 0.881±0.041 | 0.852±0.027 | 0.821±0.062
∗ The following are the full names of the dataset abbreviations: CYP2C9_SC
(CYP2C9_Substrate_CarbonMangels), Bio_Ma (Bioavailability_Ma),
SARSCoV2_VT (SARSCoV2_Vitro_Touret), CYP2D6_SC
(CYP2D6_Substrate_CarbonMangels), SARSCoV2_3CD (SARSCoV2_3CLPro_Diamond),
CYP3A4_SC (CYP3A4_Substrate_CarbonMangels), Carcinogens (Carcinogens_Lagunin),
Pgp_Bro (Pgp_Broccatelli), PAMPA (PAMPA_NCATS)
TABLE II: Average Results of 35 Tasks Training Experiments Method | Model | Mean AUCROC $\uparrow$
---|---|---
Vanilla | CNN | 0.715±0.055
GIN | 0.716±0.054
GCN | 0.734±0.051
NeuralFP | 0.736±0.049
Chemprop | 0.741±0.049
AttentiveFP | 0.75±0.063
Pretrain | KCL | 0.745±0.052
ContextPred | 0.766±0.051
AttrMasking | 0.77±0.046
| GNN-MoCE(Ours) | 0.809±0.043

####  V-A3 Baselines

We compared our proposed method against 9 common MPP models. Based on training
methods, we categorize them into two main groups: Vanilla and Pretrain.
Vanilla methods follow the classical model training approach, which involves
training on a single dataset. In contrast, Pretrain methods require additional
information sources and often leverage large-scale tasks to acquire a
foundational model. This model is then fine-tuned on the train and validation
sets of the target task and subsequently tested on the test set.

In this experiment, we employed 6 types of Vanilla methods. Among them, CNN
and NeuralFP are sequence-based methods, while the remaining 4 are graph-based
methods. The details are described as follows:

  * •

CNN [47], a basic convolutional neural network model based on SMILES
sequences, improved for MPP. This is not the graph-based method.

  * •

GCN [48], a fundamental GNN structure, which achieves message passing through
a simple aggregation function.

  * •

GIN [39], a GNN structure that introduces graph isomorphism testing
capability.

  * •

NeuralFP [5], a neural network model using autoencoders for each layer,
analyzing autoencoder error distributions during the reconstruction of the
training dataset.

  * •

Chemprop [6], a classic deep GNN structure designed for molecular embeddings,
augmented with various features that can be rapidly computed in the molecular
domain.

  * •

AttentiveFP [29], a graph neural network structure utilizing graph attention
mechanisms learned from relevant drug discovery datasets.

For Pretrain MPP methods, we primarily considered the following three pre-
training approaches. They integrate various external information into their
models.

  * •

KCL [11], a novel knowledge-enhanced contrastive learning (KCL) framework for
molecular representation learning, based on a Chemical Element Knowledge Graph
and contrastive learning. This introduces the external chemical knowledge into
MPP.

  * •

ContextPred [10], an unsupervised pre-training graph model based on context
prediction. The model undergoes pre-training on a large-scale dataset to
acquire molecular context information. This introduces supplementary molecular
structures into MPP.

  * •

AttrMasking [10], an unsupervised pre-training graph model based on masking
node attributes. The model undergoes pre-training on a large-scale dataset to
enhance node representations. This also introduces supplementary molecular
structures into MPP.

In contrast to the aforementioned baselines, our model is trained using
information across various tasks. Without utilizing additional information
sources, we integrate the training and validation sets of the existing 24
datasets. The model is jointly trained on the integrated development set and
subsequently tested on the respective test sets.

####  V-A4 Metrics

All evaluations were conducted based on AUCROC (Area Under Receiver Operating
Characteristic Curve) values. A higher value of this metric indicates better
performance.

####  V-A5 Training details

For the optimization strategy, we employed AdamW [49] with a batch size of
512, setting both the initial learning rate and weight decay to 0.01. To
dynamically adjust the learning rate, we implemented a Cosine Annealing
Schedule. The architectural composition of our model featured 60 experts in
each processing layer, and each Encoder incorporated 6 Graph Neural Networks
(GNNs), each characterized by an embedding dimension of 300. The decision
group size $k_{s}$ is set to 4, and task visiable experts number $k_{t}$ is
set to 12.

In the comparative study, we subjected the baseline models to experimentation
using their default parameters as outlined in their respective publications.
All experiments were conducted on an 80GB Nvidia A800 GPU accelerator.

###  V-B Main Experimental Results

####  V-B1 Model Performance

We evaluated the performance of the GNN-MoCE model on 35 tasks across 24
datasets and compared it with common MPP models that were separately trained
or fine-tuned on each dataset. The detailed experimental results are presented
in TABLE I, and the average performance of each model across all datasets are
presented in TABLE II. The numbers in the table represent the mean and
standard deviation of the AUC-ROC values obtained from 10 runs with different
random seeds. Higher values indicate better performance. The models are
grouped based on whether they are pretrained. In each group, they are ranked
by their average performance. The best model’s result is highlighted in bold,
and the top three models are indicated with an underscore.

We found that our model outperformed other baseline models in average
performance across all datasets and achieved the best performance in several
tasks. Notably, the datasets in TABLE I are sorted in ascending order
according to the positive class ratio (the number of positive instances
divided by the total number of instances). The results show that the tasks
with substantial gains relative to the baseline models are mostly the datasets
with limited data or extremely imbalanced class distributions.

####  V-B2 Ablation Study

To evaluate the effectiveness of our architecture, we systematically removed
the mechanisms we proposed step by step. Experiments was conducted on the ten
datasets, and their average results were presented in TABLE III. In the table,
GNN-MoCE represents the complete model, incorporating mechanisms of Expert-
Specific Projection and Expert-Specific Loss. As can be seen from the table,
compared to the GNN that does not use our method at all, our GNN-MoCE brings a
total AUCROC gain of 0.107. Specifically, the exclusion of the Expert-Specific
Loss incurs a performance loss in AUCROC of 0.041. This operation implies that
we no longer consider the dominant decision problem in the dynamic decision
group, and the expert group may degenerate into a single expert. Next, we
remove the Expert-Specific Projection and find that AUCROC decreases from
0.801 to 0.769. Of this change, 0.019 is contributed by the Attention Cosine
Loss in Expert-Specific Projection. This indicates that our approach, which
involves assigning each expert a unique perspective for observing samples and
ensuring these perspectives are as distinct as possible, has proven to be
effective.

TABLE III: Ablation Study Average Results Model | AUCROC $\uparrow$
---|---
GNN-MoCE | 0.842
GNN-MoCE w/o ES Loss | 0.801
GNN-MoCE w/o (ES Loss & $\mathcal{L}_{att}$ ) | 0.782
GNN-MoCE w/o (ES Loss & ES Projection ) | 0.769
GNN | 0.735
1 Evaluated on SkinReaction, hERG_Karim, CYP1A2_Veith
AMES, CYP2C9_Veith, CYP3A4_Veith, Pgp_Broccatelli, DILI
CYP2C19_Veith and CYP3A4_Substrate_CarbonMangels
2 ES denotes Expert-Specific
3 ES Projection = SAG + $\mathcal{L}_{att}$

###  V-C Exploratory Experimental Results

Due to the nature of our task embeddings being derived from natural language,
it is relatively simple to extend our model to new tasks. All that is required
is to modify the natural language descriptions of tasks and prepare the
corresponding datasets. To validate the scalability of this task embedding
approach, we retained the default training settings while excluding certain
tasks during training. For these tasks, we tested whether the model could
achieve reasonably good performance without any additional training, solely
relying on the natural language embeddings of the new tasks. We masked four
datasets from the training set for this purpose. Among them, CYP2C19_Veith and
CYP3A4_Veith are the most similar to the other 31 datasets, while PAMPA_NCATS
and HIA_Hou are the least similar.

The experimental results are presented in TABLE IV. In the table, column
Masked denotes the model’s performance when trained with only 31 tasks and
directly tested on these four datasets. Compared to Unmasked, which means
training and testing the model with all 35 tasks together, the performance
only drops by about 20%. When compared to the pretrained model without
finetuning, which is developed using ContextPred, our model exhibits a
performance gain ranging from 7% to 46%. This suggests that our model
possesses some ability to do OOD tasks.

TABLE IV: Results of masked tasks (AUCROC $\uparrow$) Dataset | Pretrain(ContextPred) | Masked | Unmasked
---|---|---|---
CYP3A4_Veith | 0.506 | 0.702 | 0.898
CYP2C19_Veith | 0.492 | 0.743 | 0.886
PAMPA_NCATS | 0.498 | 0.548 | 0.701
HIA_Hou | 0.486 | 0.755 | 0.953

##  VI Conclusion

In this paper, we propose the GNN-MoCE architecture to leverage the
collaborative effects among multiple experts corresponding to various MPP
tasks. The proposed method for enhancing expert collaboration is woven
throughout the entire lifecycle of the dynamic decision group. For the
generation of candidate experts, we propose the Expert-Specific Projection. By
employing the unique SAG readout parameters for each expert, we strive to
obtain distinct graph representations, thereby obtaining as diverse experts as
possible. For the formation of a dynamic decision group, the embeddings of
task natural language descriptions are utilized in routing samples, thereby
ensuring the experts involved in the collaboration are proficient in similar
tasks. As for the collaboration, the Expert-Specific Loss is introduced to
emphasize the fairness of training among the experts within the dynamic
decision group. This collaborative decision-making within the group can be
seen as a manifestation of swarm intelligence. Additionally, benefiting from
the human-editability of natural language task descriptions, our model
exhibits the potential to execute new tasks without the need for additional
training.

Considering future research, the encoder in the architecture can be replaced
with other pretrained GNN models, requiring only engineering adjustments.
Introducing high-quality pre-trained models into the encoder may further
improve generalization in OOD tasks. Besides, the proposed MoCE method can be
further generalized to other models with MoE structures, allowing MoE-
structured models to leverage the collaborative effects within the expert
group on top of their original sparse routing structure.

## Acknowledgement

This work is supported by the National Key Research and Development Program of
China under Grant No. 2022YFC3303301, the National Natural Science Foundation
of China under Grant No. 72271151, and the Blue Information Entrepreneurship
and Innovation Fund of Shanghai University of Finance and Economics.

## References

  * [1] W. P. Walters and R. Barzilay, “Applications of deep learning in molecule generation and molecular property prediction,” Accounts of chemical research, vol. 54, no. 2, pp. 263–270, 2020.
  * [2] K. Huang, T. Fu, W. Gao, Y. Zhao, Y. Roohani, J. Leskovec, C. W. Coley, C. Xiao, J. Sun, and M. Zitnik, “Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development,” arXiv preprint arXiv:2102.09548, 2021.
  * [3] R. S. Cahn, C. Ingold, and V. Prelog, “Specification of molecular chirality,” Angewandte Chemie International Edition in English, vol. 5, no. 4, pp. 385–415, 1966.
  * [4] Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel, “Gated graph sequence neural networks,” arXiv preprint arXiv:1511.05493, 2015.
  * [5] W.-H. Lee, S. Millman, N. Desai, M. Srivatsa, and C. Liu, “Neuralfp: Out-of-distribution detection using fingerprints of neural networks,” in 2020 25th International Conference on Pattern Recognition (ICPR), pp. 9561–9568, 2021.
  * [6] K. Yang, K. Swanson, W. Jin, C. Coley, P. Eiden, H. Gao, A. Guzman-Perez, T. Hopper, B. Kelley, M. Mathea, et al., “Analyzing learned molecular representations for property prediction,” Journal of chemical information and modeling, vol. 59, no. 8, pp. 3370–3388, 2019.
  * [7] Z. Guo, W. Yu, C. Zhang, M. Jiang, and N. V. Chawla, “Graseq: graph and sequence fusion learning for molecular property prediction,” in Proceedings of the 29th ACM international conference on information & knowledge management, pp. 435–443, 2020.
  * [8] J. Broberg, M. Bånkestad, and E. Ylipää, “Pre-training transformers for molecular property prediction using reaction prediction,” arXiv preprint arXiv:2207.02724, 2022.
  * [9] H. Stärk, D. Beaini, G. Corso, P. Tossou, C. Dallago, S. Günnemann, and P. Liò, “3d infomax improves gnns for molecular property prediction,” in International Conference on Machine Learning, pp. 20479–20502, PMLR, 2022.
  * [10] W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. Pande, and J. Leskovec, “Strategies for pre-training graph neural networks,” arXiv preprint arXiv:1905.12265, 2019.
  * [11] Y. Fang, Q. Zhang, H. Yang, X. Zhuang, S. Deng, W. Zhang, M. Qin, Z. Chen, X. Fan, and H. Chen, “Molecular contrastive learning with chemical element knowledge graph,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 3968–3976, 2022.
  * [12] H. Li, X. Zhao, S. Li, F. Wan, D. Zhao, and J. Zeng, “Improving molecular property prediction through a task similarity enhanced transfer learning strategy,” Iscience, vol. 25, no. 10, 2022.
  * [13] S. Liu, M. Qu, Z. Zhang, H. Cai, and J. Tang, “Multi-task learning with domain knowledge for molecular property prediction,” in NeurIPS 2021 AI for Science Workshop, 2021.
  * [14] J. Lee, I. Lee, and J. Kang, “Self-attention graph pooling,” in International conference on machine learning, pp. 3734–3743, PMLR, 2019.
  * [15] M. Orio, D. A. Pantazis, and F. Neese, “Density functional theory,” Photosynthesis research, vol. 102, pp. 443–453, 2009.
  * [16] A. Tropsha, “Best practices for qsar model development, validation, and exploitation,” Molecular informatics, vol. 29, no. 6-7, pp. 476–488, 2010.
  * [17] F. Neese, “Prediction of molecular properties and molecular spectroscopy with density functional theory: From fundamental theory to exchange-coupling,” Coordination Chemistry Reviews, vol. 253, no. 5-6, pp. 526–563, 2009.
  * [18] M. Tsubaki and T. Mizoguchi, “Fast and accurate molecular property prediction: learning atomic interactions and potentials with neural networks,” The journal of physical chemistry letters, vol. 9, no. 19, pp. 5733–5741, 2018.
  * [19] D. Weininger, “Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules,” Journal of chemical information and computer sciences, vol. 28, no. 1, pp. 31–36, 1988.
  * [20] I. Muegge and P. Mukherjee, “An overview of molecular fingerprint similarity search in virtual screening,” Expert opinion on drug discovery, vol. 11, no. 2, pp. 137–148, 2016.
  * [21] S. Wang, Y. Guo, Y. Wang, H. Sun, and J. Huang, “Smiles-bert: large scale unsupervised pre-training for molecular property prediction,” in Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informatics, pp. 429–436, 2019.
  * [22] Z. Li, M. Jiang, S. Wang, and S. Zhang, “Deep learning methods for molecular representation and property prediction,” Drug Discovery Today, p. 103373, 2022.
  * [23] G. A. Pinheiro, J. Mucelini, M. D. Soares, R. C. Prati, J. L. Da Silva, and M. G. Quiles, “Machine learning prediction of nine molecular properties based on the smiles representation of the qm9 quantum-chemistry dataset,” The Journal of Physical Chemistry A, vol. 124, no. 47, pp. 9854–9866, 2020.
  * [24] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  * [25] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Commun. ACM, vol. 60, p. 84–90, may 2017.
  * [26] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, “Neural message passing for quantum chemistry,” in International conference on machine learning, pp. 1263–1272, PMLR, 2017.
  * [27] H. Dai, B. Dai, and L. Song, “Discriminative embeddings of latent variable models for structured data,” in International conference on machine learning, pp. 2702–2711, PMLR, 2016.
  * [28] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  * [29] Z. Xiong, D. Wang, X. Liu, F. Zhong, X. Wan, X. Li, Z. Li, X. Luo, K. Chen, H. Jiang, et al., “Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism,” Journal of medicinal chemistry, vol. 63, no. 16, pp. 8749–8760, 2019.
  * [30] M. Withnall, E. Lindelöf, O. Engkvist, and H. Chen, “Building attention and edge message passing neural networks for bioactivity and physical–chemical property prediction,” Journal of cheminformatics, vol. 12, no. 1, pp. 1–18, 2020.
  * [31] B. Tang, S. T. Kramer, M. Fang, Y. Qiu, Z. Wu, and D. Xu, “A self-attention based message passing neural network for predicting molecular lipophilicity and aqueous solubility,” Journal of cheminformatics, vol. 12, no. 1, pp. 1–9, 2020.
  * [32] M. Meng, Z. Wei, Z. Li, M. Jiang, and Y. Bian, “Property prediction of molecules in graph convolutional neural network expansion,” in 2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS), pp. 263–266, IEEE, 2019.
  * [33] J. Jiang, R. Zhang, J. Ma, Y. Liu, E. Yang, S. Du, Z. Zhao, and Y. Yuan, “Trangru: focusing on both the local and global information of molecules for molecular property prediction,” Applied Intelligence, vol. 53, no. 12, pp. 15246–15260, 2023.
  * [34] R. Dey and F. M. Salem, “Gate-variants of gated recurrent unit (gru) neural networks,” in 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS), pp. 1597–1600, IEEE, 2017.
  * [35] Y. Sun, Y. Chen, W. Ma, W. Huang, K. Liu, Z. Ma, W.-Y. Ma, and Y. Lan, “Pemp: Leveraging physics properties to enhance molecular property prediction,” in Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 3505–3513, 2022.
  * [36] R. Sun, H. Dai, and A. W. Yu, “Does gnn pretraining help molecular representation?,” Advances in Neural Information Processing Systems, vol. 35, pp. 12096–12109, 2022.
  * [37] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” arXiv preprint arXiv:1701.06538, 2017.
  * [38] J. Ma, Z. Zhao, X. Yi, J. Chen, L. Hong, and E. H. Chi, “Modeling task relationships in multi-task learning with multi-gate mixture-of-experts,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’18, (New York, NY, USA), p. 1930–1939, Association for Computing Machinery, 2018.
  * [39] K. Xu, W. Hu, J. Leskovec, and S. Jegelka, “How powerful are graph neural networks?,” arXiv preprint arXiv:1810.00826, 2018.
  * [40] M. Lin, Q. Chen, and S. Yan, “Network in network,” arXiv preprint arXiv:1312.4400, 2013.
  * [41] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptive mixtures of local experts,” Neural Computation, vol. 3, no. 1, pp. 79–87, 1991.
  * [42] N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, et al., “Glam: Efficient scaling of language models with mixture-of-experts,” in International Conference on Machine Learning, pp. 5547–5569, PMLR, 2022.
  * [43] D. Eigen, M. Ranzato, and I. Sutskever, “Learning factored representations in a deep mixture of experts,” CoRR, vol. abs/1312.4314, 2013.
  * [44] E. Bengio, P.-L. Bacon, J. Pineau, and D. Precup, “Conditional computation in neural networks for faster models,” ArXiv, vol. abs/1511.06297, 2015.
  * [45] W. Hu, M. Fey, H. Ren, M. Nakata, Y. Dong, and J. Leskovec, “Ogb-lsc: A large-scale challenge for machine learning on graphs,” arXiv preprint arXiv:2103.09430, 2021.
  * [46] T. S. Ryan Greene and A. N. Lilian Weng, “New and improved embedding model,” https://openai.com/blog/new-and-improved-embedding-model, 2022.
  * [47] K. Huang, T. Fu, L. M. Glass, M. Zitnik, C. Xiao, and J. Sun, “Deeppurpose: a deep learning library for drug–target interaction prediction,” Bioinformatics, vol. 36, no. 22-23, pp. 5545–5547, 2020.
  * [48] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” arXiv preprint arXiv:1609.02907, 2016.
  * [49] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017.


September 16, 2019

# Spin-Triplet Superconductivity in UTe2 and Ferromagnetic Superconductors

Dai Aoki1,2 E-mail<EMAIL_ADDRESS>Ai Nakamura1 Fuminori Honda1 DeXin
Li1 Yoshiya Homma1 Yusei Shimizu1 Yoshiki J. Sato1 Georg Knebel2 Jean-Pascal
Brison2 Alexandre Pourret2 Daniel Braithwaite2 Gerard Lapertot2 Qun Niu2
Michal Vališka2 Hisatomo Harima3 and Jacques Flouquet2 1IMR1IMR Tohoku
University Tohoku University Oarai Oarai Ibaraki Ibaraki 311-1313 311-1313
Japan
2University Grenoble Alpes Japan
2University Grenoble Alpes CEA CEA IRIG-PHELIQS IRIG-PHELIQS F-38000 Grenoble
F-38000 Grenoble France
3Graduate School of Science France
3Graduate School of Science Kobe University Kobe University Kobe 657-8501 Kobe
657-8501 Japan Japan<EMAIL_ADDRESS>
###### Abstract

The spin-triplet state is most likely realized in uranium ferromagnetic
superconductors, UGe2, URhGe, UCoGe. The microscopic coexistence of
ferromagnetism and superconductivity means that the Cooper pair should be
realized under the strong internal field due the ferromagnetism. leading to
the spin-triplet state with equal spin pairing. The field-reinforced
superconductivity, which is observed in all three materials when the
ferromagnetic fluctuations are enhanced, is one of the strong evidences for
the spin-triplet superconductivity. We present here the results of a newly
discovered spin-triplet superconductor, UTe2, and compare those with the
results of ferromagnetic superconductors. Although no magnetic order is found
in UTe2, there are similarities between UTe2 and ferromagnetic
superconductors. For example, the huge upper critical field exceeding the
Pauli limit and the field-reentrant superconductivity for $H\parallel b$-axis
are observed in UTe2, URhGe and UCoGe. We also show the specific heat results
on UTe2 in different quality samples, focusing on the residual density of
states in the superconducting phase.

ferromagnetism, superconductivity, metamagnetism, reentrant superconductivity,
spin triplet, specific heat

The coexistence of ferromagnetism and superconductivity attracts much
attention because unconventional superconductivity with spin-triplet state is
realized. [1, 2] In general, ferromagnetism and superconductivity are
antagonistic because the large internal field due to the ferromagnetism easily
destroys the superconducting Cooper pairs in conventional superconductors.
Thus it is natural to consider that the spin-triplet superconductivity with
equal spin-pairing is realized in ferromagnetic superconductors. The
microscopic coexistence of ferromagnetism and superconductivity is established
only in uranium compounds so far, namely UGe2 [3], URhGe [4] and UCoGe [5].
All of these materials have fairly small ordered moments ($1$–$0.05\,\mu_{\rm
B}/{\rm U}$) in the ferromagnetic phase compared to that for the U free ion
($\sim 3\,\mu_{\rm B}$). Thus the $5f$ electrons for these compounds are
believed to be itinerant in the first principle, although the magnetic
anisotropy is rather strong, indicating the Ising properties. The
superconductivity occurs well below the Curie temperature, $T_{\rm Curie}$, in
the ferromagnetic state. One of the highlights in ferromagnetic
superconductors is the field-reentrant or field-reinforced superconductivity.
In URhGe, for example, the reentrant superconductivity appears when the field
is applied along the hard-magnetization $b$-axis in the orthorhombic
structure. [6] While the transition temperature $T_{\rm sc}$ is $0.25\,{\rm
K}$ at zero field, the reentrant superconducting phase has a maximum $T_{\rm
sc}$ of $0.4\,{\rm K}$ at $H_{\rm R}\sim 12\,{\rm T}$, indicating that the
superconductivity is indeed reinforced under magnetic field. The similar
field-reinforced superconductivity is also observed in UCoGe. [7]

Recently a new spin-triplet superconductor, namely UTe2 was discovered [8, 9].
UTe2 has the body-centered orthorhombic structure with the space group $Immm$
(#71, $D_{2h}^{25}$). The distance of the first nearest neighbor for U atom is
$3.78\,{\rm\AA}$, which is larger than the so-called Hill limit ($\sim
3.5\,{\rm\AA}$). Although no magnetic order was found down to $0.025\,{\rm
K}$, UTe2 is believed to be at the verge of ferromagnetic order. In fact, the
ferromagnetic fluctuations were observed in $\mu$SR [10] and NMR experiments
[11]. By substituting Te with Se, the ferromagnetic order appears at $69$ and
$33\,{\rm K}$ in UTe0.72Se1.28 and UTe0.24Se1.76, respectively, [12] although
the space group for these materials is $Pnma$, which is different from that in
UTe2. The superconducting transition occurs at $1.6\,{\rm K}$ with the sharp
and large specific heat jump. The large residual density of states nearly
$50\,{\%}$ may suggest the possibility for the spontaneous spin-polarization
and the “partially-gapped” superconductivity similar to the A1 state with non-
unitary state. However, it should be stressed that the direct transition from
the paramagnetic state to the non-unitary state at zero field is forbidden
from the symmetry restriction in this orthorhombic system. [13] Thus, the
hidden feature in the superconducting state is expected. No other transition
in the superconducting state is not observed yet at least at zero field. The
pressure study is definitely important to solve this problem. One of the
strongest support for the spin-triplet superconductivity in UTe2 is the huge
upper critical field, $H_{\rm c2}$. In all the field directions, $H_{\rm c2}$
extremely exceeds the Pauli limit ($\sim 3\,{\rm T}$) expected for the weak-
coupling BCS theory. The values of $H_{\rm c2}$ at $0\,{\rm K}$ are $7$ and
$11\,{\rm T}$ for $H\parallel a$ and $c$-axis, respectively. For $H\parallel
b$-axis, the spectacular field-reentrant superconductivity is observed. [14,
15] The transition temperature monotonously decreases with field down to
$0.4\,{\rm K}$ at $16\,{\rm T}$, then increases with field up to $0.9\,{\rm
K}$ at $35\,{\rm T}$. The first order metamagnetic transtion occurs at $H_{\rm
m}=35\,{\rm T}$ [16, 17], and the superconductivity is abruptly collapsed
above $H_{\rm m}$. The metamagnetic transition at $H_{\rm m}$ is connected to
the so-called $T_{\chi,\rm max}$ at low fields, where the magnetic
susceptibility shows a broad maximum for $H\parallel b$-axis. The magnetic
susceptibility shows the Curie-Weiss behavior at high temperatures. At low
temperatures, the anisotropic susceptibility is observed with the relation,
$\chi_{a}>\chi_{c}>\chi_{b}$, which is consistent with the anisotropy of
$H_{\rm c2}$.

In order to study more details on superconducting properties in UTe2, we have
grown single crystals of UTe2 with different quality, and measured the
specific heat at low temperatures. We compare the ($H,T$) phase diagrams for
$H\parallel b$-axis in UTe2, URhGe and UCoGe.

Figure 1: (Color online) Photographs of UTe2 single crystals grown by (a)
chemical vapor transport method and (b) Te-flux method. (c) Laue photograph of
UTe2 single crystal along $c$-axis. (d) U-Te phase diagram cited from Ref.18

Single crystals of UTe2 were grown using chemical vapor transport method. The
starting materials of U and Te were put into a quartz ampoule with the atomic
ratio, U : Te = 2 : 3, together with iodine as the transport agent to be the
density, $3\,{\rm mg/cm}^{3}$ in the inner volume of the quartz ampoule. The
ampoule was slowly heated and was kept at the temperature gradient of
$1060\,^{\circ}{\rm C}$/$1000\,^{\circ}{\rm C}$ for 10 days. Many single
crystals were obtained at lower temperature side, as shown in Fig. 1(a). The
obtained single crystals were checked by the single crystal X-ray analysis.
The lattice parameters and the atomic coordinates are in good agreement with
the values in the previous report. [19] The single crystals were oriented
using the Laue photograph, as shown in Fig. 1(c). The clear superconducting
transition was observed in resistivity and specific heat. The highest residual
resistivity ratio (RRR) is about 40. Note that the we also obtained single
crystals from the previous recipe, that is, a stoichiometric amount of
starting materials and lower temperature gradient $950\,^{\circ}{\rm
C}$/$850\,^{\circ}{\rm C}$. The single crystals were grown at high temperature
side in this case. However, the quality of the single crystal is lower with
the low RRR ($\sim 2$–$3$), and no superconductivity was observed down to
$0.1\,{\rm K}$.

As shown in Fig. 1(d), UTe2 is an incongruent melting compound in the U-Te
phase diagram, and single crystals of UTe2 can be grown using the Te-flux
method as well. The off-stoichiometric amounts of U and Te ($22$ and $78\,{\rm
at\%}$, respectively) were put into an alumina crucible, which was sealed in a
Ta-tube under Ar atmosphere gas. The Ta-tube was then sealed again in a quartz
ampoule. The quartz ampoule was slowly heated up to $1050\,^{\circ}{\rm C}$
and was cooled down to $960\,^{\circ}{\rm C}$. The Te-flux was removed at
$960\,^{\circ}{\rm C}$ in a centrifuge. The obtained single crystals were
large, as shown in Fig. 1(b). However the residual resistivity ratio is not
very large (${\rm RRR}\sim 3$). Although the superconductivity was confirmed
by the resistivity, it was not a bulk property as no anomaly was detected in
the specific heat. Hereafter, we show the results of single crystals obtained
by the chemical vapor transport method with off-stoichiometric amounts of
starting materials and the high temperature gradient.

Figure 2(a) shows the temperature dependence of the electronic specific heat
in UTe2. The part of the data is replotted from Ref. 8, 20. The data of sample
#1 show the highest $T_{\rm sc}$ with a sharp and large jump at $T_{\rm sc}$,
indicating the highest quality sample. The value of $T_{\rm sc}$ defined by
the entropy balance in #1 is $1.65\,{\rm K}$, and the residual $\gamma$-value,
$\gamma_{0}$, which is extrapolated from the fitting $C_{\rm
e}/T=\gamma_{0}+\alpha T^{2}$ at low temperatures assuming a point node gap,
is $\gamma_{0}=52\,{\rm mJ\,K^{-2}mol^{-1}}$. The residual $\gamma$-value is
equal to $44\,{\%}$ of the $\gamma$-value in the normal state. The lower
quality samples show the lower $T_{\rm sc}$ and the higher residual
$\gamma$-value. For example, $T_{\rm sc}$ and $\gamma_{0}$ in sample #4 are
$1.23\,{\rm K}$ and $89\,{\rm mJ\,K^{-2}mol^{-1}}$, respectively. In sample
#5, no superconductivity was observed down to $0.4\,{\rm K}$ in specific heat,
and it is confirmed by the resistivity measurement down $0.1\,{\rm K}$.

Figure 2(b) shows $T_{\rm sc}$ as a function of residual $\gamma$-value
normalized by the $\gamma$-value in the normal state. It is clear that $T_{\rm
sc}$ decreases with increasing the residual $\gamma$-value. It is known that
the decrease of $T_{\rm sc}$ can be described by the Abrikosov-Gor’kov pair-
breaking theory. On the basis of this model, the relation between $T_{\rm sc}$
and the residual density of states had been studied theoretically [21] and
experimentally [22] in high $T_{\rm c}$ cuprates and heavy fermion systems,
where the rapid increase of the residual density of states is reported,
compared to the decrease of $T_{\rm sc}$. This can be explained by the
unitarity scattering in unconventional superconductivity. The present result
in Fig. 2(b) supports the unconventional superconductivity in UTe2.

An important question is whether the residual density of states exists in the
ideal single crystal without impurities. In that case, the partial density of
states would be gapped, and the so-called A1 state should be realized, where
the time reversal symmetry must be broken at zero field. There is, however, no
experimental evidence for the breaking of time reversal symmetry in UTe2.

Figure 2: (Color online) (a) Electronic specific heat in the form of $C_{\rm
e}/T$ vs $T$ of UTe2 in different samples. The phonon contribution is
subtracted from the fitting at high temperature above $T_{\rm sc}$. Part of
the data is replotted from Ref. 8, 20. (b) $T_{\rm sc}$ as a function of the
normalized residual $\gamma$-value for different quality samples.

Next we show in Fig. 3 the ($H,T$) phase diagrams of UTe2 and two
ferromagnetic superconductors, URhGe and UCoGe for $H\parallel b$-axis,
corresponding to the hard-magnetization axis. The field-reentrant or field-
reinforced superconductivity is observed both in URhGe and in UCoGe. The
enhancement of superconductivity is clearly related to the suppression of
$T_{\rm Curie}$, where the ferromagnetic instabilities are realized.

In URhGe, the suppression of $T_{\rm Curie}$ leads to the spin-reorientation
at $H_{\rm R}$ in field sweep at low temperatures. The slope of magnetization
curve for $H\parallel b$-axis is larger than that for $c$-axis. The moment
gradually tilts from $c$ to $b$-axis, and finally it re-orients to $b$-axis at
$H_{\rm R}\sim 12\,{\rm T}$. The $\gamma$-value is enhanced with increasing
field, taking a maximum at $H_{\rm R}$. In NMR experiments, the spin-spin
relaxation rate, $1/T_{2}$ shows the diverging behavior around $H_{\rm R}$,
indicating the strong enhancement of the longitudinal ferromagnetic
fluctuations [23]. The 2nd order transition of $T_{\rm Curie}$ at low fields
changes into the weak 1st order transition at $H_{\rm R}$ through the
tricritical point (TCP). The reentrant superconductivity appears with the
maximum $T_{\rm sc}=0.4\,{\rm K}$ exactly at $H_{\rm R}$.

In UCoGe, the suppression of $T_{\rm Curie}$ with field is similar to the case
for URhGe. However, the spin reorientation is not observed in magnetization
curve, indicating the strong Ising property compared to URhGe. The
superconductivity shows an “S”-shaped curve, which is also connected to the
suppression of $T_{\rm Curie}$. The enhancement of $\gamma$-value and the
development of longitudinal fluctuation are observed in the field scan for
$H\parallel b$-axis.

In UTe2, the field reentrant superconductivity is also observed, while the
temperature range and field range are much wider, compared to those in
ferromagnetic superconductors, URhGe and UCoGe. The reentrant
superconductivity is again linked to the metamagnetic transition at $H_{\rm
m}$. The clear difference from ferromagnetic superconductors is that $H_{\rm
m}$ at high fields originates from the broad maximum of magnetic
susceptibility, $T_{\chi,\rm max}$, instead of $T_{\rm Curie}$. In the heavy
fermion system, it is well known that $H_{\rm m}$ is scaled with $T_{\chi,\rm
max}$ [24]. The value of $H_{\rm m}=35\,{\rm T}$ in UTe2 is consistent with
$T_{\chi,\rm max}=35\,{\rm K}$. The mass enhancement around $H_{\rm m}$ is
detected in the resistivity $A$ coefficient [16] and $\gamma$-value from the
Maxwell relation in magnetization [17] and the direct specific heat
measurements [25]. The crossover at $T_{\chi,\rm max}$ changes into the 1st
order transition at $H_{\rm m}$ through the critical end point (CEP). It
should be noted that the reentrant superconductivity is abruptly suppressed
above $H_{\rm m}$ in UTe2. On the other hand, the reentrant superconductivity
in URhGe still survives in the small field range above $H_{\rm m}$. This is
probably due to the abrupt change of $T_{\rm sc}$ as it is inferred from the
sharp increase of magnetoresistance at $H_{\rm m}$, implying the drastic
change of the electronic state at $H_{\rm m}$.

Figure 3: (Color online) ($H,T$) phase diagrams for $H\parallel b$-axis in
URhGe, UCoGe and UTe2. The data are taken from Ref. 1, 14, 16, 17

In summary, we presented the single crystal growth of the novel spin-triplet
superconductor, UTe2, and the results of specific heat in different quality
samples. The higher quality sample shows the higher $T_{\rm sc}$ and the lower
residual density of states. The rapid increase of the residual density of
states compared to the decrease of $T_{\rm sc}$ supports the unconventional
superconductivity in UTe2. The unusual field-reentrant superconductivity is a
common feature in ferromagnetic superconductors and UTe2. The precise high
field experiments from the microscopic point of views and pressure experiments
using a high quality single crystal are desired for the future studies.

## Acknowledgements

We thank Y. Tokunaga, S. Ikeda, Y. Ōnuki, K. Ishida, K. Izawa, K. Miyake, V.
Mineev, S. Ran, J. Ishizuka, Y. Yanase, K. Machida, C. Paulsen and K. Miyake
for fruitful discussion. This work was supported by ERC starting grant
(NewHeavyFermion), KAKENHI (JP15H05884, JP15H05882, JP15K21732, JP16H04006,
JP15H05745, JP19H00646), and ICC-IMR.

## References

  * [1] D. Aoki, K. Ishida, and J. Flouquet: J. Phys. Soc. Jpn. 88 (2019) 022001\.
  * [2] D. Aoki and J. Flouquet: J. Phys. Soc. Jpn. 81 (2012) 011003.
  * [3] S. S. Saxena, P. Agarwal, K. Ahilan, F. M. Grosche, R. K. W. Haselwimmer, M. J. Steiner, E. Pugh, I. R. Walker, S. R. Julian, P. Monthoux, G. G. Lonzarich, A. Huxley, I. Sheikin, D. Braithwaite, and J. Flouquet: Nature 406 (2000) 587.
  * [4] D. Aoki, A. Huxley, E. Ressouche, D. Braithwaite, J. Flouquet, J.-P. Brison, E. Lhotel, and C. Paulsen: Nature 413 (2001) 613.
  * [5] N. T. Huy, A. Gasparini, D. E. de Nijs, Y. Huang, J. C. P. Klaasse, T. Gortenmulder, A. de Visser, A. Hamann, T. Görlach, and H. v. Löhneysen: Phys. Rev. Lett. 99 (2007) 067006.
  * [6] F. Lévy, I. Sheikin, B. Grenier, and A. D. Huxley: Science 309 (2005) 1343.
  * [7] D. Aoki, T. D. Matsuda, V. Taufour, E. Hassinger, G. Knebel, and J. Flouquet: J. Phys. Soc. Jpn. 78 (2009) 113709.
  * [8] S. Ran, C. Eckberg, Q.-P. Ding, Y. Furukawa, T. Metz, S. R. Saha, I.-L. Liu, M. Zic, H. Kim, J. Paglione, and N. P. Butch: Science 365 (2019) 684\.
  * [9] D. Aoki, A. Nakamura, F. Honda, D. Li, Y. Homma, Y. Shimizu, Y. J. Sato, G. Knebel, J.-P. Brison, A. Pourret, D. Braithwaite, G. Lapertot, Q. Niu, M. Vališka, H. Harima, and J. Flouquet: J. Phys. Soc. Jpn. 88 (2019) 043702.
  * [10] S. Sundar, S. Gheidi, K. Akintola, A. M. Côté, S. R. Dunsiger, S. Ran, N. P. Butch, S. R. Saha, J. Paglione, and J. E. Sonier: arXiv:1905.06901 .
  * [11] Y. Tokunaga, H. Sakai, S. Kambe, T. Hattori, N. Higa, G. Nakamine, S. Kitagawa, K. Ishida, A. Nakamura, Y. Shimizu, Y. Homma, D. Li, F. Honda, and D. Aoki: J. Phys. Soc. Jpn. 88 (2019) 073701.
  * [12] H. Noël, M. Potel, R. Troc, and L. Shlyk: J. Solid. State. Chem. 126 (1996) 22.
  * [13] V. Mineev: private communication .
  * [14] G. Knebel, W. Knafo, A. Pourret, Q. Niu, M. Vališka, D. Braithwaite, G. Lapertot, J.-P. Brison, S. Mishra, I. Sheikin, G. Seyfarth, D. Aoki, and J. Flouquet: J. Phys. Soc. Jpn. 88 (2019) 063707.
  * [15] S. Ran, I.-L. Liu, Y. S. Eo, D. J. Campbell, P. Neves, W. T. Fuhrman, S. R. Saha, C. Eckberg, H. Kim, J. Paglione, D. Graf, J. Singleton, and N. P. Butch: arXiv:1905.04343 .
  * [16] W. Knafo, M. Vališka, D. Braithwaite, G. Lapertot, G. Knebel, A. Pourret, J.-P. Brison, J. Flouquet, and D. Aoki: J. Phys. Soc. Jpn. 88 (2019) 063705.
  * [17] A. Miyake, Y. Shimizu, Y. J. Sato, D. Li, A. Nakmura, Y. Homma, F. Honda, J. Flouquet, M. Tokunaga, and D. Aoki: J. Phys. Soc. Jpn. 88 (2019) 063706.
  * [18] Y. Xu, M. Yamazaki, and P. Villars: Jpn. J. Appl. Phys. 50 (2011) 11RH02.
  * [19] S. Ikeda, H. Sakai, D. Aoki, Y. Homma, E. Yamamoto, A. Nakamura, Y. Shiokawa, Y. Haga, and Y. Ōnuki: J. Phys. Soc. Jpn. Suppl 75 (2006) 116\.
  * [20] T. Metz, S. Bae, S. Ran, I.-L. Liu, Y. S. Eo, W. T. Fuhrman, D. F. Agterberg, S. Anlage, N. P. Butch, and J. Paglione: arXiv:1908.01069 .
  * [21] A. Okada and K. Miyake: J. Phys. Soc. Jpn. 80 (2011) 084708.
  * [22] Y. Kitaoka, K. Ishida, and K. Asayama: J. Phys. Soc. Jpn. 63 (1994) 2052\.
  * [23] Y. Tokunaga, D. Aoki, H. Mayaffre, S. Krämer, M.-H. Julien, C. Berthier, M. Horvatić, H. Sakai, S. Kambe, and S. Araki: Phys. Rev. Lett. 114 (2015) 216401.
  * [24] D. Aoki, W. Knafo, and I. Sheikin: C. R. Physique 14 (2013) 53.
  * [25] S. Imajo, Y. Kohama, A. Miyake, C. Dong, J. Flouquet, K. Kindo, and D. Aoki: J. Phys. Soc. Jpn. 88 (2019) 083705.