# Adjacency List Oriented Relational Fact Extraction
via Adaptive Multi-task Learning

Fubang Zhao1 , Zhuoren Jiang211footnotemark: 1 , Yangyang Kang1, Changlong
Sun1, Xiaozhong Liu3
1Alibaba Group, Hangzhou, China
2School of Public Affairs, Zhejiang University, Hangzhou, China
3School of Informatics, Computing and Engineering, IUB, Bloomington, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>These two authors contributed equally to this research.
Zhuoren Jiang is the corresponding author

###### Abstract

Relational fact extraction aims to extract semantic triplets from unstructured
text. In this work, we show that all of the relational fact extraction models
can be organized according to a graph-oriented analytical perspective. An
efficient model, aDjacency lIst oRiented rElational faCT (DIRECT), is proposed
based on this analytical framework. To alleviate challenges of error
propagation and sub-task loss equilibrium, DIRECT employs a novel adaptive
multi-task learning strategy with dynamic sub-task loss balancing. Extensive
experiments are conducted on two benchmark datasets, and results prove that
the proposed model outperforms a series of state-of-the-art (SoTA) models for
relational triplet extraction.

##  1 Introduction

Relational fact extraction, as an essential NLP task, is playing an
increasingly important role in knowledge graph construction Han et al. (2019);
Distiawan et al. (2019). It aims to extract relational triplet from the text.
A relational triplet is in the form of $(subject,relation,object)$ or
$(s,r,o)$ Zeng et al. (2019). While various prior models proposed for
relational fact extraction, few of them analyze this task from the perspective
of output data structure.

Figure 1: Example of exploring the relational fact extraction task from the
perspective of directed graph representation method as output data structure.

As shown in Figure 1, the relational fact extraction can be characterized as a
directed graph construction task, where graph representation flexibility and
heterogeneity accompany additional benefaction. In practice, there are three
common ways to represent graphs Gross and Yellen (2005):

Edge List is utilized to predict a sequence of triplets (edges). The recent
sequence-to-sequence based models, such as NovelTagging Zheng et al. (2017),
CopyRE Zeng et al. (2018), CopyRL Zeng et al. (2019), and PNDec Nayak and Ng
(2020), fall into this category.

Edge list is a simple and space-efficient way to represent a graph Arifuzzaman
and Khan (2015). However, there are three problems. First, the triplet
overlapping problem Zeng et al. (2018). For instance, as shown in Figure 1,
for triplets (Obama, nationality, USA) and (Obama, president_of, USA), there
are two types of relations between the “Obama” and “USA”. If the model only
generates one sequence from the text Zheng et al. (2017), it may fail to
identify the multi-relation between entities. Second, to overcome the triplet
overlapping problem, the model may have to extract the triplet element
repeatedly Zeng et al. (2018), which will increase the extraction cost. Third,
there could be an ordering problem Zeng et al. (2019): for multiple triplets,
the extraction order could influence the model performance.

Adjacency Matrices are used to predict matrices that represent exactly which
entities (vertices) have semantic relations (edges) between them. Most early
works, which take a pipeline approach Zelenko et al. (2003); Zhou et al.
(2005), belong to this category. These models first recognize all entities in
text and then perform relation classification for each entity pair. The
subsequent neural network-based models Bekoulis et al. (2018); Dai et al.
(2019), that attempt to extract entities and relations jointly, can also be
classified into this category.

Compared to edge list, adjacency matrices have better relation (edge)
searching efficiency Arifuzzaman and Khan (2015). Furthermore, adjacency
matrices oriented models is able to cover different overlapping cases Zeng et
al. (2018) for relational fact extraction task. But the space cost of this
approach can be expensive. For most cases, the output matrices are very
sparse. For instance, for a sentence with $n$ tokens, if there are $m$ kinds
of relations, the output space is $n\cdot n\cdot m$, which can be costly for
graph representation efficiency. This phenomenon is also illustrated in Figure
1.

Adjacency List is designed to predict an array of linked lists that serves as
a representation of a graph. As depicted in Figure 1, in the adjacency list,
each vertex $v$ (key) points to a list (value) containing all other vertices
connected to $v$ by several edges. Adjacency list is a hybrid graph
representation between edge list and adjacency matrices Gross and Yellen
(2005), which can balance space and searching efficiency111More detailed
complexity analyses of different graph representations are provided in
Appendix section 6.3.. Due to the structural characteristic of the adjacency
list, this type of model usually adopts a cascade fashion to identify subject,
object, and relation sequentially. For instance, the recent state-of-the-art
model CasRel Wei et al. (2020) can be considered as an exemplar. It utilizes a
two-step framework to recognize the possible object(s) of a given subject
under a specific relation. However, CasRel is not fully adjacency list
oriented: in the first step, it use $subject$ as the key; while in the second
step, it predicts $(relation,object)$ pairs using adjacency matrix
representation.

Despite its considerable potential, the cascade fashion of adjacency list
oriented model may cause problems of sub-task error propagation Shen et al.
(2019), i.e., errors from ancestor sub-tasks may accumulate to threaten
downstream ones, and sub-tasks can hardly share supervision signals. Multi-
task learning Caruana (1997) can alleviate this problem, however, the sub-task
loss balancing problem Chen et al. (2018); Sener and Koltun (2018) could
compromise its performance.

Based on the analysis from the perspective of output data structure, we
propose a novel solution, aDjacency lIst oRiented rElational faCT extraction
model (DIRECT), with the following advantages:

$\bullet$ For efficiency, DIRECT is a fully adjacency list oriented model,
which consists of a shared BERT encoder, the Pointer-Network based subject and
object extractors, and a relation classification module. In Section 3.4, we
provide a detailed comparative analysis222Theoretical representation
efficiency analysis of graph representative models are described in Appendix
section 6.4. to demonstrate the efficiency of the proposed method.

$\bullet$ From the performance viewpoint, to address sub-task error
propagation and sub-task loss balancing problems, DIRECT employs a novel
adaptive multi-task learning strategy with the dynamic sub-task loss balancing
approach. In Section 3.2 and 3.3, the empirical experimental results
demonstrate DIRECT can achieve the state-of-the-art performance of relational
fact extraction task, and the adaptive multi-task learning strategy did play a
positive role in improving the task performance.

The major contributions of this paper can be summarized as follows:

1\. We refurbish the relational fact extraction problem by leveraging an
analytical framework of graph-oriented output structure. To the best of our
knowledge, this is a pioneer investigation to explore the output data
structure of relational fact extractions.

2\. We propose a novel solution, DIRECT333To help other scholars reproduce the
experiment outcome, we will release the code and datasets via GitHub:
https://github.com/fyubang/direct-ie., which is a fully adjacency list
oriented model with a novel adaptive multi-task learning strategy.

3\. Through extensive experiments on two benchmark datasets3, we demonstrate
the efficiency and efficacy of DIRECT. The proposed DIRECT outperforms the
state-of-the-art baseline models.

##  2 The DIRECT Framework

Figure 2: An overview of the proposed DIRECT framework

In this section, we will introduce the framework of the proposed DIRECT model,
which includes a shared BERT encoder and three output layers: subject
extraction, object extraction, and relation classification. As shown in Figure
2, DIRECT is fully adjacency list oriented. The input sentence is firstly fed
into the subject extraction module to extract all subjects. Then each
extracted subject is concatenated with the sentence, and fed into the object
extraction module to extract all objects, which can form a set of subject-
object pairs. Finally, the subject-object pair is concatenated with sentence,
and fed into the relation classification module to get the relations between
them. For balancing the weights of sub-task losses and to improve the global
task performance, three modules share the BERT encoder layer and are trained
with an adaptive multi-task learning strategy.

###  2.1 Shared BERT Encoder

In the DIRECT framework, the encoder is used to extract the semantic features
from the inputs for three modules. As aforementioned, we employ the BERT
Devlin et al. (2019) as the shared encoder to make use of its pre-trained
knowledge and attention mechanism.

The architecture of the shared method is shown in Figure 2. The lower
embedding layer and transformers Vaswani et al. (2017) are shared across all
the three modules, while the top layers represent the task-speciﬁc outputs.

The encoding process is as follows:

$\mathbf{h}^{t}=\text{BERT}(\mathbf{x}^{t})$ (1)

where $\mathbf{x}^{t}=[w_{1},...,w_{n}]$ is the input text of task $t$ and
$\mathbf{h}^{t}$ is the hidden vector sequence of the input. Due to the
limited space, the detailed architecture of BERT please refer to the original
paper Devlin et al. (2019).

###  2.2 Subject and Object Extraction

The subject and object extraction modules are motivated by the Pointer-Network
Vinyals et al. (2015) architecture, which are widely used in Machine Reading
Comprehension (MRC) Rajpurkar et al. (2016) task. Different from MRC task that
only needs to extract a single span, the subject and object extractions need
to extract multiple spans. Therefore, in the training phase, we replace
$softmax$ function with $sigmoid$ function for the activation function of the
output layer, and replace cross entropy (CE) Goodfellow et al. (2016) with
binary cross entropy (BCE) Luc et al. (2016) for the loss function.
Specifically, we will perform independent binary classifications for each
token twice to indicate whether the current token is the start or the end of a
span. The probability of a token to be start or end is as follows:

$\displaystyle p_{i,{\text{start}}}^{t}$
$\displaystyle=\sigma(\mathbf{W}_{\text{start}}^{t}\cdot
h_{i}+\mathbf{b}_{\text{start}}^{t})$ (2) $\displaystyle
p_{i,{\text{end}}}^{t}$ $\displaystyle=\sigma(\mathbf{W}_{\text{end}}^{t}\cdot
h_{i}+\mathbf{b}_{\text{end}}^{t})$ (3)

where $h_{i}$ represents the hidden vector of the $i_{th}$ token, $t\in[s,o]$
represents subject and object extraction respectively,
$\mathbf{W}^{t}\in\mathbb{R}^{h\times 1}$ represents the trainable weight,
$\mathbf{b}^{t}\in\mathbb{R}^{1}$ is the bias and $\sigma$ is $sigmoid$
function.

During inference, we first recognize all the start positions by checking if
the probability $p_{i,{\text{start}}}^{t}>\alpha$, where $\alpha$ is the
threshold of extraction. Then, we identify the corresponding end position with
the largest probability $p_{i,{\text{end}}}^{t}$ between two neighboring start
positions. Concretely, assuming $\text{pos}_{j,\text{start}}$ is the start
position of the $j_{th}$ span, the corresponding end position is:

$\text{pos}_{j,\text{end}}=\underset{\text{pos}_{j,\text{start}}<=i<\text{pos}_{j+1,\text{start}}}{\mathrm{argmax}}p_{i,{\text{end}}}^{t}$
(4)

Though the overall structure is similar, the inputs for subject and object
extraction are different. When extracting the subject, only the original
sentence needs to be input:

$\displaystyle\mathbf{x}$ $\displaystyle=[w_{1},...,w_{n}]$ (5)
$\displaystyle\text{input}^{s}$
$\displaystyle=[\text{[cls]},\mathbf{x},\text{[sep]}]$ (6)

where $w_{i}$ represents the $i_{th}$ token of the original sentence.

Meanwhile, the object extraction is based on the corresponding subject. To
form the input, the subject $s$ and the original sentence $x$ are concatenated
with $[sep]$ as follows:

$\text{input}^{o}=[\text{[cls]},s,\text{[sep]},\mathbf{x},\text{[sep]}]$ (7)

###  2.3 Relation classification

The output layer of relation classification is relatively simple, which is a
normal multi-label classification model. The $[cls]$ vector obtained by BERT
encoder is used as the sentence embedding. A fully connected layer is used for
the nonlinear transformation, and perform multi-label classification to
predict relations of the input subject-object pair. The detailed operations of
relation classification are as follows:

$\mathbf{P}^{r}=\sigma(\mathbf{W}^{r}\cdot h_{\text{[cls]}}+\mathbf{b}^{r})$
(8)

where $\mathbf{P}^{r}\in\mathbb{R}^{c}$ is the predicted probability vector of
relations, $\sigma$ is $sigmoid$ function,
$\mathbf{W}^{r}\in\mathbb{R}^{h\times c}$ and
$\mathbf{b}^{r}\in\mathbb{R}^{c}$ are the trainable weights and bias, $h$ is
the hidden size of encoder, $c$ is the number of relations, and
$h_{\text{[cls]}}$ denotes the hidden vector of the first token [cls]. The
input for relation classification task is as follows:

$\text{input}^{r}=[\text{[cls]},s,\text{[sep]},o,\text{[sep]},\mathbf{x},\text{[sep]}]$
(9)

Initialize model parameters $\Theta$ randomly;

Load pre-trained BERT parameters for shared encoder;

Prepare the data for each task $t$ and pack them into mini-batch: $D^{t}$,
$t\in[s,o,r]$ ;

Get the number of batch for each task: $n^{t}$;

Set the number of epoch for training: $epoch_{max}$;

for  _ $epoch$ in $1,2,...,epoch_{max}$_ do

1\. Merge all the datasets: $D=D^{s}\cup D^{o}\cup D^{r}$;

2\. Shuffle D;

3\. Initialize EMA for each task $v^{t}=1$ and its decay $\epsilon=0.99$ ;

for  _$b^{t}$ in $D$_ do

// $b^{t}$ is a mini-batch of $D^{t}$ ;

4\. Compute loss: $l^{t}(\Theta)$ ;

5\. Update EMA: $v^{t}=(1-\epsilon)\cdot\sum(l^{t})+\epsilon\cdot v^{t}$ ;

6\. Calculate and normalize the weights: $w^{t}=(v^{t}/n^{t})/(v^{r}/n^{r})$ ;

7\. Update model $\Theta$ with gradient: $\nabla(w^{t}\cdot\bar{l^{t}})$ ;

end for

end for

Algorithm 1 Adaptive Multi-task Learning with Dynamic Loss Balancing

###  2.4 Adaptive Multi-task Learning

In DIRECT, subject extraction module, object extraction module, and relation
classification module can be considered as three sub-tasks. As aforementioned,
if we train each module directly and separately, the error propagation problem
would reduce the task performance. Meanwhile, three independent encoders would
consume more memory. Therefore, we use multi-task learning to alleviate this
problem, and the encoder layer is shared across three modules.

However, applying multi-task learning could be challenging in DIRECT, due to
the following problems:

$\bullet$ The input and output of the three modules are different, which means
we cannot simply sum up the loss of each task.

$\bullet$ How should we balance the weights of losses for three sub-task
modules?

These issues can affect the final results of multi-task training Shen et al.
(2019); Sener and Koltun (2018).

In this work, based on the architecture of MT-DNN Liu et al. (2019b), we
propose a novel adaptive multi-task learning strategy to address the above
problems. The algorithm is shown as Algorithm 1. Basically, the datasets are
firstly split into mini-batches. A batch is then randomly sampled to calculate
the loss. The parameters of the shared encoder and its task-specific layer are
updated accordingly. Especially, the learning effect of each task $t$ is
different and dynamically changing during training. Therefore, an approach of
adaptively adjusting the weights of task losses is applied. The sum of sub-
task’s loss $\sum l^{t}$ is utilized to approximate its optimization effect.
The adaptive weight adjusting strategy ensures that the more room a sub-task
has to be optimized, the more weight its loss will receive. Furthermore, an
exponential moving average (EMA) Lawrance and Lewis (1977) is maintained to
avoid the drastic fluctuations of loss weights. Last but not least, to make
sure that each task has enough influence on the shared encoder, the weight of
the sub-task will be penalized according to the training data amount of each
sub-task.

##  3 Experiments

###  3.1 Dataset and Experiment Setting

Datasets. Two public datasets are used for evaluation: NYT Riedel et al.
(2010) is originally produced by the distant supervision approach. There are
1.18M sentences with 24 predeﬁned relation types in NYT. WebNLG Gardent et al.
(2017) is originally created for Natural Language Generation (NLG) tasks. Zeng
et al. (2018) adopts this dataset for relational triplet extraction task. It
contains 246 predeﬁned relation types. There are different versions of these
two datasets. To facilitate comparison evaluation, we use the datasets
released by Zeng et al. (2018) and follow their data split rules.

Besides the basic relational triplet extraction, recent studies are focusing
on the relational triplet overlapping problem Zeng et al. (2018); Wei et al.
(2020). Follow the overlapping pattern definition of relational triplets Zeng
et al. (2018), the sentences in both datasets are divided into three
categories, namely, Normal, EntityPairOverlap (EPO), and SingleEntityOverlap
(SEO). The statistics of the two datasets are described in Table 1.

Category | NYT | WebNLG
---|---|---
Train | Test | Train | Test
Normal | 37013 | 3266 | 1596 | 246
EPO | 9782 | 978 | 227 | 26
SEO | 14735 | 1297 | 3406 | 457
ALL | 56195 | 5000 | 5019 | 703
Table 1: Statistics of Dataset NYT and WebNLG. Note that a sentence can belong
to both EPO class and SEO class.

Baselines: the following strong state-of-the-art (SoTA) models have been
compared in the experiments.

$\bullet$ NovelTagging Zheng et al. (2017) introduces a tagging scheme that
transforms the joint entity and relation extraction task into a sequence
labeling problem. It can be considered as edge list oriented.

$\bullet$ CopyRE Zeng et al. (2018) is a seq2seq based model with the copy
mechanism, which can effectively extract overlapping triplets. It has two
variants: CopyRE${}_{\text{one}}$ employs one decoder; CopyRE${}_{\text{mul}}$
employs multiple decoders. CopyRE is also edge list oriented.

$\bullet$ GraphRel Fu et al. (2019) is a GCN (graph convolutional networks)
Kipf and Welling (2017) based model, where a relation-weighted GCN is utilized
to learn the interaction between entities and relations. It is a two phases
model: GraphRel${}_{\text{1p}}$ denotes 1st-phase extraction model;
GraphRel${}_{\text{2p}}$ denotes full extraction model. GraphRel is adjacency
matrices oriented.

$\bullet$ CopyRL Zeng et al. (2019) combines the reinforcement learning with a
seq2seq model to automatically learn the extraction order of triplets. CopyRL
is edge list oriented.

$\bullet$ CasRel Wei et al. (2020) is a cascade binary tagging framework,
where all possible subjects are identiﬁed in the ﬁrst stage, and then for each
identiﬁed subject, all possible relations and the corresponding objects are
simultaneously identiﬁed by a relation speciﬁc tagger. This work recently
achieves the SoTA results. As aforementioned, CasRel is partially adjacency
list oriented.

Evaluation Metrics: following the previous work Zeng et al. (2018); Wei et al.
(2020), different models are compared by using standard micro Precision
(Prec.), Recall (Rec.), and F1-score444In this study, the results of baseline
models are all self-reported results from their original papers. Meanwhile,
the experimental results of our proposed model are the average of five runs..
An extracted relational triplet (subject, relation, object) is regarded as
correct only if the relation and the heads of both subject and object are all
correct.

Implementation Details. The hyper-parameters are determined on the validation
set. To avoid the evaluation bias, all reported results from our method are
averaged results for 5 runs. More implementation details are described in
Appendix section 6.1.

Method | Category | NYT | WebNLG
---|---|---|---
Prec. | Rec. | F1 | Prec. | Rec. | F1
NovelTaggingZheng et al. (2017) | EL | 62.4 | 31.7 | 42.0 | 52.5 | 193. | 28.3
$\text{CopyRE}_{\text{One}}$Zeng et al. (2018) | EL | 59.4 | 53.1 | 56.0 | 32.2 | 28.9 | 30.5
$\text{CopyRE}_{\text{Mul}}$Zeng et al. (2018) | EL | 61.0 | 56.6 | 58.7 | 37.7 | 36.4 | 37.1
$\text{GraphRel}_{\text{1p}}$Fu et al. (2019) | AM | 62.9 | 57.3 | 60.0 | 42.3 | 39.2 | 40.7
$\text{GraphRel}_{\text{2p}}$Fu et al. (2019) | AM | 63.9 | 60.0 | 61.9 | 44.7 | 41.1 | 42.9
CopyRLZeng et al. (2019) | EL | 77.9 | 67.2 | 72.1 | 63.3 | 59.9 | 61.6
CasRelWei et al. (2020) | AL${}_{\text{P}}$ | 89.7 | 89.5 | 89.6 | 93.4 | 90.1 | 91.8
DIRECT(Ours) | AL${}_{\text{F}}$ | 92.3$\pm$0.32 | 92.8$\pm$0.26 | 92.5$\pm$0.09 | 93.6$\pm$0.1 | 92.7$\pm$0.24 | 93.2$\pm$0.07
Table 2: Results of different methods on NYT and WebNLG datasets. EL: Edge
List; AM: Adjacency Matrices; AL${}_{\text{P}}$: Adjacency List (Partially);
AL${}_{\text{F}}$: Adjacency List (Fully).

###  3.2 Results and Analysis

Relational Triplet Extraction Performance. The task performances on two
datasets are summarized in Table 2. Based on the experiment results, we have
the following observations and discussions:

$\bullet$ The proposed DIRECT model outperformed all baseline models in terms
of all evaluation metrics on both datasets, which proved DIRECT model can
effectively address the relational triplet extraction task.

$\bullet$ The best-performed model (DIRECT) and runner-up model (CasRel) were
both adjacency list oriented model. These two models overwhelmingly
outperformed other models, which indicated the considerable potential of
adjacency list (as the output data structure) for improving the task
performance.

Method | Element | NYT | WebNLG
---|---|---|---
CasRel | s | 93.5 | 95.7
o | 93.5 | 95.3
r | 94.9 | 94.0
DIRECT(Ours) | s | 95.4 | 97.3
o | 96.4 | 96.4
r | 97.8 | 97.4
Table 3: F1-score for extracting elements of relational triplets on NYT and
WebNLG datasets.

$\bullet$ To further compare the relation extraction ability of DIRECT and
CasRel, we took a closer look at the extraction performance of relational
triplet elements from these two models. As shown in Table 3555More detailed
results with Precision and Recall are provided in Appendix section 6.2.,
DIRECT outperformed CasRel in terms of all relational triplet elements on both
datasets. These empirical results suggested that, for relational triplet
extraction, a fully adjacency list oriented model (DIRECT) may have advantages
over a partially oriented one (CasRel).

Figure 3: F1 score of extracting relational triples from sentences with
different overlapping patterns on NYT dataset.

Ability in Handling The Overlapping Problem. The relational facts in sentences
are often complicated. Different relational triplets may have overlaps in a
sentence. To verify the ability of our models in handling the overlapping
problem, we conducted further experiments on NYT dataset. Figure 3 illustrated
of F1 scores of extracting relational triplets from sentences with different
overlapping patterns. DIRECT outperformed all baseline models in terms of all
overlapping patterns. These results demonstrated the effectiveness of the
proposed model in solving the overlapping problem.

Ability in Handling Multiple Relation Extraction. We further compared the
model’s ability of extracting relations from sentences that contain multiple
triplets. The sentences in NYT and WebNLG were divided into 5 categories. Each
category contained sentences that had 1,2,3,4 or $\geq$ 5 triplets. The
triplet number was denoted as $N$. As shown in Table 4:

$\bullet$ DIRECT achieved the best performance for all triplet categories on
both datasets. These experimental results demonstrated our model had an
excellent ability in handling multiple relation extraction.

$\bullet$ In both NYT and WebNLG datasets, when the sentences contained more
triplets, the leading advantage of DIRECT became greater. This observation
indicated that DIRECT was good at solving complex relational fact extraction.

Method | NYT | WebNLG
---|---|---
$N=1$ | $N=2$ | $N=3$ | $N=4$ | $N\geq 5$ | $N=1$ | $N=2$ | $N=3$ | $N=4$ | $N\geq 5$
Count | 3244 | 1045 | 312 | 291 | 108 | 268 | 174 | 128 | 89 | 44
$\text{CopyRE}_{\text{One}}$ | 66.6 | 52.6 | 49.7 | 48.7 | 20.3 | 65.2 | 33.0 | 22.2 | 14.2 | 13.2
$\text{CopyRE}_{\text{Mul}}$ | 67.1 | 58.6 | 52.0 | 53.6 | 30.0 | 59.2 | 42.5 | 31.7 | 24.2 | 30.0
$\text{GraphRel}_{\text{1p}}$ | 69.1 | 59.5 | 54.4 | 53.9 | 37.5 | 63.8 | 46.3 | 34.7 | 30.8 | 29.4
$\text{GraphRel}_{\text{2p}}$ | 71.0 | 61.5 | 57.4 | 55.1 | 41.1 | 66.0 | 48.3 | 37.0 | 32.1 | 32.1
CopyRL | 71.7 | 72.6 | 72.5 | 77.9 | 45.9 | 63.4 | 62.2 | 64.4 | 57.2 | 55.7
CasRel | 88.2 | 90.3 | 91.9 | 94.2 | 83.7 | 89.3 | 90.8 | 94.2 | 92.4 | 90.9
DIRECT(Ours) | 90.4 | 93.1 | 94.3 | 95.8 | 93.1 | 90.3 | 92.8 | 94.8 | 94.0 | 92.9
Table 4: F1-score of extracting relational triplets from sentences with
different number (denoted as N) of triplets.

###  3.3 Ablation Study

To validate the effectiveness of components in DIRECT, We implemented several
model variants for ablation tests666Due to the length limitation, we list two
main ablation experiments, the rest will be provided in the Appendix section
6.2.. The results of the comparison on NYT dataset are shown in Table 5. In
particular, we aim to address the following two research questions:

RQ1: Is it possible to improve the model performance by sharing the parameters
of extraction layers?

RQ2: Did the proposed adaptive multi-task learning strategy improve the task
performance?

Method | NYT
---|---
Prec. | Rec. | F1
DIRECT${}_{\text{shared}}$ | 92.1 | 91.6 | 91.9
DIRECT${}_{\text{equal}}$ | 90.6 | 91.3 | 91.0
DIRECT | 92.3 | 92.8 | 92.5
Table 5: Results of model variants for ablation tests.

Effects of Sharing Extraction Layer Parameters (RQ1). As described in Section
2, the structures of subject extraction and object extraction output layers
are exactly the same. To answer RQ1, we merged the subject extraction and
object extraction layers into one entity extraction layer by sharing the
parameters of output layers of these two modules, denoted as DIRECTshared.
From the results of Table 5, we can observe that, sharing the parameters of
output layers of two extraction modules would reduce the performance of the
model. A possible explanation is that, although the output of these two
modules is similar, the semantics of subject and object are different. Hence,
directly sharing the output parameters of two modules could lead to an
unsatisfactory performance.

Effects of Adaptive Multi-task Learning (RQ2). As described in Section 2, the
adaptive multi-task learning strategy with the dynamic sub-task loss balancing
approach is proposed for improving the task performance. To answer RQ2, we
replaced the adaptive multi-task learning strategy with an ordinary learning
strategy. In this strategy, the losses of three sub-tasks were computed with
equal weights, denoted as DIRECTequal. From the results of Table 5, we can
observe that, by using adaptive multi-task learning, DIRECT was able to get a
1.5 percentage improvement on the F1-score. This significant improvement
indicated that adaptive multi-task learning played a positive role in the
balance of sub-task learning and can improve the global task performance.

Method | Category | NYT | WebNLG
---|---|---|---
CopyRe | EL | 329 | 712
MHS | AM | 57369 | 26518
CasRel | AL${}_{\text{P}}$ | 3084 | 15836
DIRECT | AL${}_{\text{F}}$ | 238 | 542
Table 6: Graph representation efficiency estimation based on the predicted
logits amount. EL: Edge List; AM: Adjacency Matrices; AL${}_{\text{P}}$:
Adjacency List (Partially); AL${}_{\text{F}}$: Adjacency List (Fully).

###  3.4 Graph Representation Efficiency Analysis

Based on the amount estimation of predicted logits777Numeric output $(0/1)$ of
the last layer, we conduct a graph representation efficiency analysis to
demonstrate the efficiency of the proposed method888From the graph
representation perspective, when a method requires fewer logits to represent
the graph (set of triples), it will reduce the model fitting difficulty..

For each graph representation category, we choose one representative
algorithms. Edge List: CopyRE Zeng et al. (2018); Adjacency Matrices: MHS
Bekoulis et al. (2018); Adjacency List: CasRel (partially) Wei et al. (2020)
and the proposed DIRECT (fully).

The averaged predicted logits estimation for one sample999The theoretical
analysis of predicted logits for different models are described in Appendix
section 6.4. of different models on two datasets are shown in Table 6. MHS is
adjacency matrices oriented, it has the most logits that need to be predicted.
Since CasRel is partially adjacency list oriented, it needs to predict more
logits than DIRECT. Theoretically, as an edge list oriented, the predicted
logits of CopyRE should be the least. But, as described in Section 1, it needs
to extract the entities repeatedly to handle the overlapping problem. Hence,
its graph representation efficiency could be worse than our model. The
structure of our model is simple and fully adjacency list oriented. Therefore,
from the viewpoint of predicted logits estimation, DIRECT is the most
representative-efficient model.

##  4 Related Work

Relation Fact Extraction. In this work, we show that all of the relational
fact extraction models can be unified into a graph-oriented output structure
analytical framework. From the perspective of graph representation, the prior
models can be divided into three categories. Edge List, this type of model
usually employs sequence-to-sequence fashion, such as NovelTagging Zheng et
al. (2017), CopyRE Zeng et al. (2018), CopyRL Zeng et al. (2019), and PNDec
Nayak and Ng (2020). Some models of this category may suffer from the triplet
overlapping problem and expensive extraction cost. Adjacency Matrices, many
early pipeline approaches Zelenko et al. (2003); Zhou et al. (2005); Mintz et
al. (2009) and recent neural network-based models Bekoulis et al. (2018); Dai
et al. (2019); Fu et al. (2019), can be classified into this category. The
main problem for this type of model is the graph representation efficiency.
Adjacency List, the recent state-of-the-art model CasRel Wei et al. (2020) is
a partially adjacency list oriented model. In this work, we propose DIRECT
that is a fully adjacency list oriented relational fact extraction model. To
the best of our knowledge, few previous works analyze this task from the
output data structure perspective. GraphRel Fu et al. (2019) employs a graph-
based approach, but it is utilized from an encoding perspective, while we
analyze it from the perspective of output structure. Our work is a pioneer
investigation to analyze the output data structure of relational fact
extraction.

Multi-task Learning. Multi-task Learning (MTL) can improve the model
performance. Caruana (1997) summarizes the goal succinctly: “it improves
generalization by leveraging the domain-specific information contained in the
training signals of related task.” It has two benefits Vandenhende et al. :
(1) multiple tasks share a single model, which can save memory. (2) Associated
tasks complement and constrain each other by sharing information, which can
reduce overfitting and improve global performance. There are two main types of
MTL: hard parameter sharing Baxter (1997) and soft parameter sharing Duong et
al. (2015). Most of the multi-task learning is done by summing the loses
directly, this approach is not suitable for our case. When the input and
output are different, it is impossible to get two losses in one forward
propagation. MT-DNN Liu et al. (2019b) is proposed for this problem.
Furthermore, MTL is difficult for training, the magnitudes of different task-
losses are different, and the direct summation of losses may lead to a bias
for a particular task. There are already some studies proposed to address this
problem Chen et al. (2018); Guo et al. (2018); Liu et al. (2019a). They all
try to dynamically adjust the weight of the loss according to the magnitude of
the loss, the difficulty of the problem, the speed of learning, etc. In this
study, we adopt MT-DNN’s framework, and propose an adaptive multi-task
learning strategy that can dynamically adjust the loss weight based on the
averaged EMA Lawrance and Lewis (1977) of the training data amount, task
difficulty, etc.

##  5 Conclusion

In this paper, we introduce a new analytical perspective to organize the
relational fact extraction models and propose DIRECT model for this task.
Unlike existing methods, DIRECT is fully adjacency list oriented, which
employs a novel adaptive multi-task learning strategy with dynamic sub-task
loss balancing. Extensive experiments on two public datasets, prove the
efficiency and efficacy of the proposed methods.

## Acknowledgments

We are thankful to the anonymous reviewers for their helpful comments. This
work is supported by Alibaba Group through Alibaba Research Fellowship
Program, the National Natural Science Foundation of China (61876003), the Key
Research and Development Plan of Zhejiang Province (Grant No.2021C03140), the
Fundamental Research Funds for the Central Universities, and Guangdong Basic
and Applied Basic Research Foundation (2019A1515010837).

## References

  * Arifuzzaman and Khan (2015) Shaikh Arifuzzaman and Maleq Khan. 2015.  Fast parallel conversion of edge list to adjacency list for large-scale graphs.  In _Proceedings of the Symposium on High Performance Computing_ , pages 17–24.
  * Baxter (1997) Jonathan Baxter. 1997.  A bayesian/information theoretic model of learning to learn via multiple task sampling.  _Machine learning_ , 28(1):7–39.
  * Bekoulis et al. (2018) Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018.  Joint entity recognition and relation extraction as a multi-head selection problem.  _Expert Systems with Applications_ , 114:34–45.
  * Caruana (1997) Rich Caruana. 1997.  Multitask learning.  _Machine learning_ , 28(1):41–75.
  * Chen et al. (2018) Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018.  Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks.  In _International Conference on Machine Learning_ , pages 794–803. PMLR.
  * Dai et al. (2019) Dai Dai, Xinyan Xiao, Yajuan Lyu, Shan Dou, Qiaoqiao She, and Haifeng Wang. 2019\.  Joint extraction of entities and overlapping relations using position-attentive sequence labeling.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 6300–6308.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Distiawan et al. (2019) Bayu Distiawan, Gerhard Weikum, Jianzhong Qi, and Rui Zhang. 2019.  Neural relation extraction for knowledge base enrichment.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 229–240.
  * Duong et al. (2015) Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. 2015.  Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser.  In _Proceedings of the 53rd annual meeting of the Association for Computational Linguistics and the 7th international joint conference on natural language processing (volume 2: short papers)_ , pages 845–850.
  * Fu et al. (2019) Tsu-Jui Fu, Peng-Hsuan Li, and Wei-Yun Ma. 2019.  Graphrel: Modeling text as relational graphs for joint entity and relation extraction.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1409–1418.
  * Gardent et al. (2017) Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017.  Creating training corpora for nlg micro-planners.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 179–188.
  * Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016.  _Deep learning_ , volume 1.  MIT press Cambridge.
  * Gross and Yellen (2005) Jonathan L Gross and Jay Yellen. 2005.  _Graph theory and its applications_.  CRC press.
  * Guo et al. (2018) Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. 2018.  Dynamic task prioritization for multitask learning.  In _European Conference on Computer Vision_ , pages 282–299. Springer.
  * Han et al. (2019) Xu Han, Tianyu Gao, Yuan Yao, Deming Ye, Zhiyuan Liu, and Maosong Sun. 2019.  Opennre: An open and extensible toolkit for neural relation extraction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations_ , pages 169–174.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations (ICLR)_.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-supervised classification with graph convolutional networks.  In _International Conference on Learning Representations (ICLR)_.
  * Lawrance and Lewis (1977) AJ Lawrance and PAW Lewis. 1977.  An exponential moving-average sequence and point process (ema1).  _Journal of Applied Probability_ , pages 98–113.
  * Liu et al. (2019a) Shikun Liu, Edward Johns, and Andrew J Davison. 2019a.  End-to-end multi-task learning with attention.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1871–1880.
  * Liu et al. (2019b) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019b.  Multi-task deep neural networks for natural language understanding.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4487–4496.
  * Luc et al. (2016) Pauline Luc, Camille Couprie, Soumith Chintala, and Jakob Verbeek. 2016.  Semantic segmentation using adversarial networks.  _arXiv preprint arXiv:1611.08408_.
  * Mintz et al. (2009) Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009.  Distant supervision for relation extraction without labeled data.  In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_ , pages 1003–1011.
  * Nayak and Ng (2020) Tapas Nayak and Hwee Tou Ng. 2020.  Effective modeling of encoder-decoder architecture for joint entity and relation extraction.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8528–8535.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392.
  * Riedel et al. (2010) Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010.  Modeling relations and their mentions without labeled text.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , pages 148–163. Springer.
  * Sener and Koltun (2018) Ozan Sener and Vladlen Koltun. 2018.  Multi-task learning as multi-objective optimization.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , pages 525–536.
  * Shen et al. (2019) Tao Shen, Xiubo Geng, QIN Tao, Daya Guo, Duyu Tang, Nan Duan, Guodong Long, and Daxin Jiang. 2019.  Multi-task learning for conversational question answering over a large-scale knowledge base.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2442–2451.
  * (28) Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool.  Multi-task learning for dense prediction tasks: A survey.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , pages 6000–6010.
  * Vinyals et al. (2015) Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015.  Pointer networks.  In _Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 2_ , pages 2692–2700.
  * Wei et al. (2020) Zhepei Wei, Jianlin Su, Yue Wang, Yuan Tian, and Yi Chang. 2020.  A novel cascade binary tagging framework for relational triple extraction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1476–1488.
  * Zelenko et al. (2003) Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2003.  Kernel methods for relation extraction.  _Journal of machine learning research_ , 3(Feb):1083–1106.
  * Zeng et al. (2020) Daojian Zeng, Haoran Zhang, and Qianying Liu. 2020.  Copymtl: Copy mechanism for joint extraction of entities and relations with multi-task learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 9507–9514.
  * Zeng et al. (2019) Xiangrong Zeng, Shizhu He, Daojian Zeng, Kang Liu, Shengping Liu, and Jun Zhao. 2019\.  Learning the extraction order of multiple relational facts in a sentence with reinforcement learning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 367–377.
  * Zeng et al. (2018) Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao. 2018.  Extracting relational facts by an end-to-end neural model with copy mechanism.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 506–514.
  * Zhang et al. (2020) Ranran Haoran Zhang, Qianying Liu, Aysa Xuemo Fan, Heng Ji, Daojian Zeng, Fei Cheng, Daisuke Kawahara, and Sadao Kurohashi. 2020.  Minimize exposure bias of seq2seq models in joint entity and relation extraction.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 236–246.
  * Zheng et al. (2017) Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. 2017.  Joint extraction of entities and relations based on a novel tagging scheme.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1227–1236.
  * Zhou et al. (2005) GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005.  Exploring various knowledge in relation extraction.  In _Proceedings of the 43rd annual meeting of the association for computational linguistics (acl’05)_ , pages 427–434.

##  6 Appendix

###  6.1 Implementation Details

We adopted the pre-trained BERT model [BERT-Base-Cased]101010Available at:
https://storage.googleapis.com/bert_models/
2018_10_18/cased_L-12_H-768_A-12.zip as our encoder, where the number of
Transformer layers was 12 and the hidden size was 768. The token types of
input were always set to 0.

We used Adam as our optimizer and applied a triangular learning rate schedule
as suggested by original BERT paper. In addition, we adopted a lazy mechanism
for optimization. Different from the momentum mechanism of ordinary Adam
optimizer Kingma and Ba (2015) that updated the output layer parameters for
all tasks, this lazy-Adam mechanism wouldn’t update the parameters of non-
current tasks.

The dacay rate $\epsilon$ of EMA was set to 0.99 as default. The max sequence
length was 128.

The other hyper-parameters were determined on the validation set. Notably,
considering our special decoding strategy, we raised the threshold of
extraction to 0.9 to balance the precision and the recall. The threshold of
relation classification was set to 0.5 as default. The hyper-parameter setting
was listed in Table 7.

Our mthod were implemented by Pytorch111111https://pytorch.org/ and run on a
server configured with a Tesla V100 GPU, 16 CPU, and 64G memory.

Hyper-parameter | NYT | WebNLG
---|---|---
Learning Rate | 8e-5 | 1e-4
Epoch Num. | 15 | 60
Batch Size | 32 | 16
Table 7: Hyper-parameter setting for NYT and WebNLG datasets.

###  6.2 Supplementary Experimental Results

Ablation Study. To validate the effectiveness of components in DIRECT, We
implemented several model variants for ablation tests respectively. For
experimental fairness, we kept the other components in the same settings when
modifying one module.

  * •

DIRECTshared, we merged the subject extraction and object extraction layers
into one entity extraction layer by sharing the parameters of output layers of
these two modules.

  * •

DIRECTequal, we replaced the adaptive multi-task learning strategy with an
ordinary learning strategy. In this strategy, the losses of three sub-tasks
were computed with equal weights, denoted as DIRECTequal.

  * •

DIRECTthreshold, we simply recognized all the start and end positions of
entities by checking if the probability $p_{i,{\text{start/end}}}^{t}>\alpha$,
where $\alpha$ was the threshold of extraction.

  * •

DIRECTadam, we used ordinary Adam as optimizer.

Method | NYT
---|---
Prec. | Rec. | F1
DIRECT${}_{\text{shared}}$ | 92.1 | 91.6 | 91.9
DIRECT${}_{\text{equal}}$ | 90.6 | 91.3 | 91.0
DIRECT${}_{\text{threshold}}$ | 92.8 | 92.0 | 92.4
DIRECT${}_{\text{adam}}$ | 92.1 | 92.9 | 92.5
DIRECT | 92.9 | 92.1 | 92.5
Table 8: Results of model variants for ablation tests. Method | Element | NYT | WebNLG
---|---|---|---
Prec. | Rec. | F1 | Prec. | Rec. | F1
CasRel | s | 94.6 | 92.4 | 93.5 | 98.7 | 92.8 | 95.7
o | 94.1 | 93.0 | 93.5 | 97.7 | 93.0 | 95.3
r | 96.0 | 93.8 | 94.9 | 96.6 | 91.5 | 94.0
Ours | s | 95.1 | 95.1 | 95.1 | 97.1 | 96.8 | 96.9
o | 97.2 | 96.3 | 96.7 | 96.4 | 96.3 | 96.3
r | 98.6 | 98.3 | 98.5 | 97.6 | 97.3 | 97.4
Table 9: Results on extracting elements of relational triplets Method | NYT
---|---
Prec. | Rec. | F1
$\text{MHS}^{*}$ Bekoulis et al. (2018) | 60.7 | 58.6 | 59.6
$\text{CopyMTL}_{one}$Zeng et al. (2020) | 72.7 | 69.2 | 70.9
$\text{CopyMTL}_{mul}$Zeng et al. (2020) | 75.7 | 68.7 | 72.0
WDec Nayak and Ng (2020) | 88.1 | 76.1 | 81.7
PNDec Nayak and Ng (2020) | 80.6 | 77.3 | 78.9
Seq2UMTree Zhang et al. (2020) | 79.1 | 75.1 | 77.1
DIRECT(ours) | 90.2 | 90.2 | 90.2
Table 10: Results of different methods under Exact-Match Metrics. * marks
results reproduced by official implementation.

From the results of Table 8, we can observe that:

  1. 1.

Sharing the parameters of output layers of subject and object extraction
modules would reduce the performance of the model.

  2. 2.

Compared to ordinary multi-task learning strategy, by using adaptive multi-
task learning, DIRECT was able to get a 1.5 percentage point improvement on
F1-score.

  3. 3.

There would be a slight drop in performance, if we just used a simple
threshold policy to recognize the start and end positions of an entity.

  4. 4.

Despite the difference in precision and recall, there was no significant
difference between these two optimizers (ordinary-Adam & lazy-Adam ) for the
task.

Results on Extracting Elements of Relational Triplets. The complete extraction
performance of relational triplet elements from DIRECT and CaslRel are listed
in Table 9. DIRECT outperformed CasRel in terms of all relational triplet
elements on both datasets. These empirical results suggest that, for
relational triplet extraction, a fully adjacency list oriented model (DIRECT)
may have advantages over a partially oriented one (CasRel).

Category | Method | Theoretical | NYT | WebNLG
---|---|---|---|---
Edge List | CopyRe | $4kl+kr$ | 329 | 712
Adjacency Matrices | MHS | $llr$ | 57369 | 26518
Adjacency List (Partially) | CasRel | $2l+2slr$ | 3084 | 15836
Adjacency List (Fully) | DIRECT | $2l+2sl+or$ | 238 | 542
Table 11: Graph representation efficiency based on the theoretical logits
amount and the estimated logits amount on two benchmark datasets.

Results of Different Methods under Exact-Match Metrics. In experiment section,
we followed the match metric from Zeng et al. (2018), which only required to
match the first token of entity span. Many previous works adopted this match
metric Fu et al. (2019); Zeng et al. (2019); Wei et al. (2020).

In fact, our model is capable of extracting the complete entities. Therefore,
we collected papers that reported the results of exact-match metrics
(requiring to match the complete entity span). The following strong state-of-
the-art (SoTA) models have been compared:

$\bullet$ CopyMTL Zeng et al. (2020) is a multi-task learning framework, where
conditional random ﬁeld is used to identify entities, and a seq2seq model is
adopted to extract relational triplets.

$\bullet$ WDec Nayak and Ng (2020) fuses a seq2seq model with a new
representation scheme, which enables the decoder to generate one word at a and
can handle full entity names of different length and overlapping entities.

$\bullet$ PNDec Nayak and Ng (2020) is a modiﬁcation of seq2seq model. Pointer
networks are used in the decoding framework to identify the entities in the
sentence using their start and end locations.

$\bullet$ Seq2UMTree Zhang et al. (2020) is a modiﬁcation of seq2seq model,
which employs an unordered-multi-tree decoder to to minimize exposure bias.

The task performances on NYT dataset are summarized in Table 10. The proposed
DIRECT model outperformed all baseline models in terms of all evaluation
metrics. This experimental results further confirmed the efficacy of DIRECT
for relational fact extraction task.

###  6.3 Complexity Analysis of Graph Representations

For a graph $G=(V,E)$, $|V|$ denotes the number of nodes/entities and $|E|$
denotes the number of edges/relations. Suppose there are $m$ kinds of
relations, $d(v)$ denotes the number of edges from node $v$.

$\bullet$ Edge List Complexity

  1. $-$

Space: $O(|E|)$

  2. $-$

Find all edges/relations from a node: $O(|E|)$

$\bullet$ Adjacency Matrices Complexity

  1. $-$

Space: $O(|V|\cdot|V|\cdot m)$

  2. $-$

Find all edges/relations from a node: $O(|V|\cdot m)$

$\bullet$ Adjacency List Complexity

  1. $-$

Space: $O(|V|+|E|)$

  2. $-$

Find all edges/relations from a node: $O(d(v))$

###  6.4 Graph Representation Efficiency Analysis

Based on the amount estimation of predicted logits121212Numeric output of the
last layer $(0/1)$, we conduct a graph representation efficiency analysis to
demonstrate the efficiency of proposed method131313As aforementioned, from the
graph representation perspective, when a method requires fewer logits to
represent the graph (set of triples), it will reduce the model fitting
difficulty..

For each graph representation category, we choose one representative model
algorithms. Edge List: CopyRE Zeng et al. (2018); Adjacency Matrices: MHS
Bekoulis et al. (2018); Adjacency List: CasRel (partially) Wei et al. (2020)
and DIRECT (fully).

Formally, for a sentence whose length is $l$ ($l$ tokens), there are $r$ types
of relations, $k$ denotes the number of triplets. Suppose there are $s$ keys
(subjects) and $o$ values (corresponding amount of object-based lists) in
adjacency list. The theoretical logits amount and the estimated logits amount
on two benchmark datasets (NYT and WebNLG) are shown in Table 11. From the
viewpoint of predicted logits estimation, DIRECT is the most representative-
efficient model.


# DESI Constraints on Exponential Quintessence

Omar F. Ramadan<EMAIL_ADDRESS>Department of Physics $\&$ Astronomy,
University of Hawai‘i, Watanabe Hall, 2505 Correa Road, Honolulu, HI, 96822,
USA  Jeremy Sakstein<EMAIL_ADDRESS>Department of Physics $\&$
Astronomy, University of Hawai‘i, Watanabe Hall, 2505 Correa Road, Honolulu,
HI, 96822, USA  David Rubin<EMAIL_ADDRESS>Department of Physics $\&$
Astronomy, University of Hawai‘i, Watanabe Hall, 2505 Correa Road, Honolulu,
HI, 96822, USA

###### Abstract

The DESI collaboration have recently analyzed their first year of data,
finding a preference for thawing dark energy scenarios when using
parameterized equations of state for dark energy. We investigate whether this
preference persists when the data is analyzed within the context of a well-
studied field theory model of thawing dark energy, exponential quintessence.
No preference for this model over $\Lambda$CDM is found, and both models are
poorer fits to the data than the Chevallier-Polarski-Linder $w_{0}$–$w_{a}$
parameterization. We demonstrate that the worse fit is due to a lack of sharp
features in the potential that results in a slowly-evolving dark energy
equation of state that does not have enough freedom to simultaneously fit the
combination of the supernovae, DESI, and cosmic microwave background data. Our
analysis provides guidance for constructing dynamical dark energy models that
are able to better accommodate the data.

The origin of the present day acceleration of the cosmic expansion, dark
energy (DE), remains a mystery, even after a quarter of a century of research.
Previously, all observations were compatible with dark energy driven by a
cosmological constant $\Lambda$, but this has recently been challenged by the
DESI first year data release [1], which, when analyzed in combination with the
Planck and ACT cosmic microwave background (CMB) measurements and Type Ia
supernovae data, either PantheonPlus [2], Union3 [3], or DESY5 [4], shows a
preference for thawing dark energy at the level of $2.5\sigma$, $3.5\sigma$,
and $3.9\sigma$ respectively. In this scenario, the equation of state (EOS) of
dark energy $w(z)$ was frozen at a constant value in the past but recently
began to evolve away from this, in contrast to $\Lambda$ which has constant
$w(z)=-1$. The thawing DE preference manifests when the data is fit to the
Chevallier-Polarski-Linder $w_{0}$–$w_{a}$ parameterization [5, 6], which is a
phenomenological relation:

$w(z)=w_{0}+w_{a}\frac{z}{1+z}$ (1)

with $w_{0}$ and $w_{a}$ free parameters that are fit to the data. DESI report
$w_{0}=-0.727\pm 0.067$ and $w_{a}=-1.05^{+0.31}_{-0.27}$ using CMB+DESI+DESY5
datasets.

While parameterizations such as $\eqref{eq:CPL}$ are helpful for
characterizing the data and as consistency tests of the null $\Lambda$CDM
hypothesis, they do not provide any interpretation of data within the context
of fundamental physics, motivating investigations of the degree to which
competing microphysical models of dark energy can accommodate the data. In
this work, we explore the implications of the first DESI data release for a
quintessence model of thawing dark energy, exponential quintessence.

In quintessence models [7, 8, 9, 10, 11], dark energy is driven by a scalar
field $\phi$ with mass $m$ that is initially frozen at its initial condition
by Hubble friction so that $w=-1$ but begins to roll sometime in the recent
past when $H\sim m$. This rolling causes the EOS to deviate from $-1$ with
$w\geq-1$. The specific action we consider is

$S=\int\mathrm{d}^{4}x\sqrt{-g}\left[\frac{M_{\rm
Pl}^{2}}{2}R(g)-\frac{1}{2}\partial_{\mu}\phi\partial^{\mu}\phi-V(\phi)\right]$
(2)

where matter is minimally coupled to the metric $g_{\mu\nu}$ and $M_{\rm
Pl}^{2}=(8\pi G)^{-1}$ is the reduced Planck mass. In a Friedmann-Lemaître-
Robertson-Walker (FLRW) universe, the scalar behaves as a perfect fluid with
density parameter and equation of state

$\displaystyle w_{\phi}$
$\displaystyle=\frac{\dot{\phi}^{2}-2V(\phi)}{\dot{\phi}^{2}+2V(\phi)}$ (3)
$\displaystyle\Omega_{\phi}$ $\displaystyle=\frac{\dot{\phi}^{2}}{6H^{2}M_{\rm
Pl}^{2}}+\frac{V(\phi)}{3H^{2}M_{\rm Pl}^{2}}.$ (4)

The evolution of the scalar is determined by the Klein-Gordon equation

$\ddot{\phi}+3H\dot{\phi}+\frac{\mathrm{d}V}{\mathrm{d}\phi}=0.$ (5)

Equations (3)–(5) elucidate how quintessence fields can behave as thawing dark
energy. At early times, when $z\gg 1$, the field has initial condition
$\phi=\phi_{i}$ with mass $m^{2}=V^{\prime\prime}(\phi_{i})$. Provided that
$m\ll H$, the friction term $(3H\dot{\phi}\sim H^{2}\phi)$ will dominate over
the restoring term $(V^{\prime}(\phi)\sim m^{2}\phi)$ and the field will be
frozen at $\phi_{i}$. According to (3) and (4), the field behaves a
cosmological constant with EOS $w_{\phi}\approx-1$. As the universe expands,
$H$ decreases, reaching $H\sim m$ around $z\sim 1$. At this point, the field
begins to roll or thaw, gaining kinetic energy so that $w_{\phi}>-1$. The
current phase of dark energy corresponds to the scalar slowly-rolling down its
potential.

The phenomenology of quintessence DE depends upon the choice of potential. In
this work, we will study the exponential quintessence model

$V(\phi)=V_{0}e^{-\lambda\frac{\phi}{M_{\rm Pl}}},$ (6)

an archetypal potential that arises generically in beyond the Standard Model
theories such as string theory and supergravity [12, 13, 14]. Despite the non-
linearity of equation (5) and the Fridemann equations, the solution space of
exponential quintessence is well-understood because the equations can be
written in an autonomous form, implying that dynamical systems methods can be
used to identify the steady-state solutions [9, 10, 11, 15]. The system of
equations admits a dark energy dominated global attractor with
$\Omega_{\phi}=1$ and

$w_{\phi}=-1+\frac{\lambda^{2}}{3}$ (7)

provided that $\lambda<\sqrt{3}$. The thawing DE scenario can then be realized
within this potential as follows. At early times, the field is frozen such
that $w_{\phi}\approx-1$ but the field thaws and begins to roll to this
attractor at $z\sim 1$. The current phase of thawing DE corresponds to the
approach to this attractor. The attractor cannot be reached at the present day
because this would imply a DE-dominated universe in conflict with
observations, and would not match the DESI predictions because $w$ is constant
at the attractor. This introduces some sensitivity to the initial conditions.

We now test this scenario against the DESI data by fitting it to the
combination of CMB+DESI+Union3. The CMB data include Planck 2018 CMB spectra
[16], CMB gravitational lensing from a combination of Planck 2020 lensing [17,
18] and ACT DR6 [19, 20]. This is the same combination of data used by DESI.
We implemented the exponential potential into CLASS [21, 22] to evolve the
cosmology and used the Cobaya [23] framework to sample using the Markov Chain
Monte Carlo (MCMC) [24, 25] algorithm. Convergence was deemed to be achieved
when the standard Gelman-Rubin criteria $R-1<0.01$ [26] was met. To analyze
our chains and plotting, we made use of GetDist [27]. The initial conditions
were chosen using the following considerations. For the initial field,
$\phi_{i}$, we made use of a symmetry of the model:
$\phi\rightarrow\phi+\phi_{0}$, $V_{0}\rightarrow
V_{0}\exp({\lambda{\phi_{0}}/{M_{\rm Pl}}})$ where $\phi_{0}$ is a constant,
which allowed us to fix $\phi_{i}$ to an arbitrary value without loss of
generality. We chose $\phi_{i}=-4.583M_{\rm Pl}$. For the initial field
velocity $\dot{\phi}_{i}$, we used attractor initial conditions. At early
times, the field is approximately frozen so we assumed slow-roll and set
$\ddot{\phi}=0$ in Eq. (5) yielding $\dot{\phi_{i}}=\frac{\lambda
V_{0}}{3H_{i}}\exp({-\lambda{\phi_{i}}/{M_{\rm Pl}}})$. We modified CLASS to
shoot for $V_{0}$ such that $V_{0}=3H_{0}^{2}M_{\rm Pl}^{2}\Omega_{\phi}$ in
order to close the universe. We also fit the $w_{0}$–$w_{a}$ parameterization
to the same data.

Parameter & Model |  Flat $\boldsymbol{\Lambda}$CDM |  $\boldsymbol{w_{a}w_{a}}$CDM |  $\boldsymbol{w_{\phi}}$CDM
---|---|---|---
Sampled Parameters |  |  |
$\log(10^{10}A_{\mathrm{s}})$ | $3.053(3.059)^{+0.013}_{-0.014}$ | $3.040(3.040)\pm 0.013$ | $3.056(3.051)\pm 0.013$
$n_{\mathrm{s}}$ | $0.9681(0.9688)\pm 0.0036$ | $0.9657(0.9668)\pm 0.0038$ | $0.9691(0.9692)\pm 0.0037$
$\Omega_{b}h^{2}$ | $0.02245(0.02247)\pm 0.00013$ | $0.02238(0.02242)\pm 0.00014$ | $0.02248(0.02249)\pm 0.00014$
$\Omega_{c}h^{2}$ | $0.11876(0.11856)\pm 0.00084$ | $0.11968(0.11982)\pm 0.00097$ | $0.11840(0.11839)\pm 0.00089$
$100\theta_{*}$ | $1.04199(1.04193)\pm 0.00028$ | $1.04187(1.04185)\pm 0.00029$ | $1.04202(1.04199)\pm 0.00029$
$\tau_{\mathrm{reio}}$ | $0.0590(0.0614)\pm 0.0071$ | $0.0526(0.0529)\pm 0.0072$ | $0.0608(0.0588)^{+0.0070}_{-0.0084}$
$w_{0}$ | $\cdots$ | $-0.656(-0.679)\pm 0.099$ | $\cdots$
$w_{a}$ | $\cdots$ | $-1.22(-1.14)^{+0.42}_{-0.34}$ | $\cdots$
$\lambda$ | $\cdots$ | $\cdots$ | $0.60(0.74)^{+0.38}_{-0.27}$
Derived Parameters |  |  |
$H_{0}\left[\text{km/s/Mpc}\right]$ | $67.92(67.98)\pm 0.39$ | $66.52(66.61)\pm 0.94$ | $66.92(66.613)^{+0.99}_{-0.77}$
$\Omega_{m}$ | $0.3075(0.3065)\pm 0.0051$ | $0.3227(0.3221)\pm 0.0095$ | $0.3162(0.3189)^{+0.0070}_{-0.010}$
$w_{\phi}$ | $\cdots$ | $\cdots$ | $-0.936(-0.919)^{+0.038}_{-0.064}$
$\boldsymbol{\chi^{2}}$ statistics |  |  |
$\chi^{2}_{\rm bf}(\Delta)$ | $2835.45$ | $2822.10(-13.5)$ | $2832.87(-2.58)$
$\chi^{2}_{\rm bf}/{\rm DoF}$ | $1.21$ | $1.21$ | $1.21$
Tension Level | $\cdots$ | $3.02\sigma$ | $1.24\sigma(\text{n.s.})$

Table 1:  Marginalized posteriors for flat $\Lambda$CDM, $w_{0}w_{a}$CDM, and
quintessence models using CMB+DESI+Union3 datasets, showing the mean (best-
fit) and the $68\%$ confidence interval where the $\Lambda$CDM parameters
share the same prior across models. We also show the best-fitting
$\chi^{2}_{bf}(\Delta)$, where $\Delta=\chi^{2}_{\rm bf,model}-\chi^{2}_{\rm
bf,\Lambda\text{CDM}}$ represents the difference between the best-fitting
$\chi^{2}$ values with respect to $\Lambda$CDM. The level of tension with
$\Lambda$CDM are reported in the final row with “n.s.” indicating an
insignificant tension.

Our results are given in table 1, with 2D contours and marginalized posteriors
shown in Figure 2. We reproduce the DESI result that the data prefer
$w_{0}$–$w_{a}$ over $\Lambda$CDM at $\sim 3\sigma$, but find no statistically
significant preference for exponential quintessence. We therefore conclude
that both $\Lambda$CDM and exponential quintessence are disfavored compared
with $w_{0}$–$w_{a}$.

Figure 1: The equation of state for the best-fitting exponential quintessence
model ($w_{\phi}$, black) and the CLP parameterization (red). The dashed line
corresponds to the $\Lambda$CDM model with $w_{\Lambda}=-1$, and the dotted
line represents $w_{0}+w_{a}$, which is the asymptotic EOS for the CLP
parameterization for $z\gg 0$. We also show the marginalized posteriors for
the EOS today for both models with the combination of CMB+DESI+Union3 datasets
at the $68\%$ level.

The reason for this can be seen in figure 1 where we plot $w(z)$ for the best-
fitting $w_{0}$–$w_{a}$ and exponential quintessence models. Both models have
$w(z)>-1$ at the present time and decreasing towards more negative values in
the past, but the $w_{0}$–$w_{a}$ model is able to reach $w=-1$ in a shorter
time. As discussed by DESI [1] and further investigated by [28], the DESI
preference for thawing dark energy is driven by low-redshift anomalies in the
supernovae and DESI BAO data. The higher redshift DESI points are consistent
with $\Lambda$CDM. $w_{0}$–$w_{a}$ accommodates this by having $w_{0}>-1$ and
a large negative value of $w_{a}$ to ensure a rapid return $w\approx-1$. The
increasingly negative values at larger redshifts are not problematic because
DE is subdominant at this time and the model behaves similarly to
$\Lambda$CDM. In contrast, the EOS for the exponential model varies less
rapidly because the field is slowly-rolling. The EOS only tends to $w=-1$ at
higher redshifts when DE is sub-dominant, so the model is unable to
accommodate each data point as well. This suggests that quintessence
potentials with sharper features e.g., hill-top or plateau models may be able
to better-fit the data because they allow for more rapid variations in $w(z)$
around the onset of DE. Indeed, reference [29] drew an identical conclusion
using a different method where they determined an equivalent $w_{0}$-$w_{a}$
parameterization for three classes of quintessence models, finding that
exponential models lie outside the DESI $1\sigma$ contours but that hill-top
and plateau models are compatible.

Interpreting the data to identify the microphysics of dark energy remains a
paramount goal of cosmology, and our results have helped to elucidate the
requisite features that quintessence models must incorporate in order to
accommodate the DESI data. There are several avenues for followup
investigations. First, fitting other proposed quintessence potentials to the
data would help to identify the best-fitting models; and, second, one could
look at more general scalar field models of dark energy such as coupled
quintessence [30, 31, 32], k-essence [33, 34, 35], multi-field models [36],
and modified gravity [37, 38, 39, 40]. One could also go beyond scalar field
models e.g., [41, 42, 43]. Our investigation suggests that any such models
must allow for a sufficiently steep change in $w(z)$ around $z\sim 0.5$. In
addition to the hill-top and plateau quintessence models above, we note that
models such as symmetron dark energy [44, 45] that use phase transitions to
start a scalar field rolling possess such features, as do models where
relativistic species decouple around $z\sim 1$ and inject energy into a scalar
such as mass-varying neutrino models [46, 47, 48, 49], among others.

Figure 2: Marginalized posteriors for the cosmological models studied in this
work using CMB+DESI+Union3 data. The inner contours denote the $68\%$
confidence level (CL), while the outer contours denote the $95\%$ CL. Both
$w_{0}w_{a}$CDM and quintessence models encompass the $\Lambda$CDM limit.

Note Added: While this manuscript was in preparation, reference [50], which
also studies exponential quintessence in light of the DESI data, appeared on
the arXiv. Our results agree with theirs.

Acknowledgements: The technical support and advanced computing resources from
University of Hawai‘i Information Technology Services – Cyberinfrastructure,
funded in part by the National Science Foundation CC awards #2201428 and
#2232862 are gratefully acknowledged.

## References

  * Adame  _et al._ [2024] A. G. Adame _et al._ (DESI), (2024), arXiv:2404.03002 [astro-ph.CO] .
  * Brout  _et al._ [2022] D. Brout _et al._ , Astrophys. J. 938, 110 (2022), arXiv:2202.04077 [astro-ph.CO] .
  * Rubin  _et al._ [2023] D. Rubin _et al._ , (2023), arXiv:2311.12098 [astro-ph.CO] .
  * Abbott  _et al._ [2024] T. M. C. Abbott _et al._ (DES), (2024), arXiv:2401.02929 [astro-ph.CO] .
  * Chevallier and Polarski [2001] M. Chevallier and D. Polarski, Int. J. Mod. Phys. D 10, 213 (2001), arXiv:gr-qc/0009008 .
  * Linder [2003] E. V. Linder, Phys. Rev. Lett. 90, 091301 (2003), arXiv:astro-ph/0208512 .
  * Ratra and Peebles [1988] B. Ratra and P. J. E. Peebles, Phys. Rev. D 37, 3406 (1988).
  * Caldwell  _et al._ [1998] R. R. Caldwell, R. Dave, and P. J. Steinhardt, Phys. Rev. Lett. 80, 1582 (1998), arXiv:astro-ph/9708069 .
  * Copeland  _et al._ [1998] E. J. Copeland, A. R. Liddle, and D. Wands, Phys. Rev. D 57, 4686 (1998), arXiv:gr-qc/9711068 .
  * Copeland  _et al._ [2006] E. J. Copeland, M. Sami, and S. Tsujikawa, Int. J. Mod. Phys. D 15, 1753 (2006), arXiv:hep-th/0603057 .
  * Bahamonde  _et al._ [2018] S. Bahamonde, C. G. Böhmer, S. Carloni, E. J. Copeland, W. Fang, and N. Tamanini, Phys. Rept. 775-777, 1 (2018), arXiv:1712.03107 [gr-qc] .
  * Wetterich [1995] C. Wetterich, Astron. Astrophys. 301, 321 (1995), arXiv:hep-th/9408025 .
  * Binetruy [1999] P. Binetruy, Phys. Rev. D 60, 063502 (1999), arXiv:hep-ph/9810553 .
  * Barreiro  _et al._ [2000] T. Barreiro, E. J. Copeland, and N. J. Nunes, Phys. Rev. D 61, 127301 (2000), arXiv:astro-ph/9910214 .
  * Ramadan  _et al._ [2024] O. F. Ramadan, T. Karwal, and J. Sakstein, Phys. Rev. D 109, 063525 (2024), arXiv:2309.08082 [astro-ph.CO] .
  * Aghanim  _et al._ [2020] N. Aghanim _et al._ (Planck), Astron. Astrophys. 641, A6 (2020), [Erratum: Astron.Astrophys. 652, C4 (2021)], arXiv:1807.06209 [astro-ph.CO] .
  * Akrami  _et al._ [2020] Y. Akrami _et al._ (Planck), Astron. Astrophys. 643, A42 (2020), arXiv:2007.04997 [astro-ph.CO] .
  * Carron  _et al._ [2022] J. Carron, M. Mirmelstein, and A. Lewis, JCAP 09, 039 (2022), arXiv:2206.07773 [astro-ph.CO] .
  * Qu  _et al._ [2024] F. J. Qu _et al._ (ACT), Astrophys. J. 962, 112 (2024), arXiv:2304.05202 [astro-ph.CO] .
  * Madhavacheril  _et al._ [2024] M. S. Madhavacheril _et al._ (ACT), Astrophys. J. 962, 113 (2024), arXiv:2304.05203 [astro-ph.CO] .
  * Blas  _et al._ [2011] D. Blas, J. Lesgourgues, and T. Tram, JCAP 07, 034 (2011), arXiv:1104.2933 [astro-ph.CO] .
  * Lesgourgues [2011] J. Lesgourgues, (2011), arXiv:1104.2932.
  * Torrado and Lewis [2021] J. Torrado and A. Lewis, JCAP 05, 057 (2021), arXiv:2005.05290.
  * Hastings [1970] W. K. Hastings, Biometrika 57, 97 (1970).
  * Metropolis  _et al._ [1953] N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller, The journal of chemical physics 21, 1087 (1953).
  * Gelman and Rubin [1992] A. Gelman and D. B. Rubin, Statist. Sci. 7, 457 (1992).
  * Lewis [2019] A. Lewis, (2019), arXiv:1910.13970 [astro-ph.IM] .
  * Colgáin  _et al._ [2024] E. O. Colgáin, M. G. Dainotti, S. Capozziello, S. Pourojaghi, M. M. Sheikh-Jabbari, and D. Stojkovic, (2024), arXiv:2404.08633 [astro-ph.CO] .
  * Shlivko and Steinhardt [2024] D. Shlivko and P. Steinhardt, (2024), arXiv:2405.03933 [astro-ph.CO] .
  * Amendola [2000] L. Amendola, Phys. Rev. D 62, 043511 (2000), arXiv:astro-ph/9908023 .
  * Sakstein [2015] J. Sakstein, Phys. Rev. D 91, 024036 (2015), arXiv:1409.7296 [astro-ph.CO] .
  * Sakstein and Verner [2015] J. Sakstein and S. Verner, Phys. Rev. D 92, 123005 (2015), arXiv:1509.05679 [gr-qc] .
  * Chiba  _et al._ [2000] T. Chiba, T. Okabe, and M. Yamaguchi, Phys. Rev. D 62, 023511 (2000), arXiv:astro-ph/9912463 .
  * Armendariz-Picon  _et al._ [2000] C. Armendariz-Picon, V. F. Mukhanov, and P. J. Steinhardt, Phys. Rev. Lett. 85, 4438 (2000), arXiv:astro-ph/0004134 .
  * Armendariz-Picon  _et al._ [2001] C. Armendariz-Picon, V. F. Mukhanov, and P. J. Steinhardt, Phys. Rev. D 63, 103510 (2001), arXiv:astro-ph/0006373 .
  * Cicoli  _et al._ [2020] M. Cicoli, G. Dibitetto, and F. G. Pedro, JHEP 10, 035 (2020), arXiv:2007.11011 [hep-th] .
  * Clifton  _et al._ [2012] T. Clifton, P. G. Ferreira, A. Padilla, and C. Skordis, Phys. Rept. 513, 1 (2012), arXiv:1106.2476 [astro-ph.CO] .
  * Joyce  _et al._ [2015] A. Joyce, B. Jain, J. Khoury, and M. Trodden, Phys. Rept. 568, 1 (2015), arXiv:1407.0059 [astro-ph.CO] .
  * Koyama [2016] K. Koyama, Rept. Prog. Phys. 79, 046902 (2016), arXiv:1504.04623 [astro-ph.CO] .
  * Ferreira [2019] P. G. Ferreira, Ann. Rev. Astron. Astrophys. 57, 335 (2019), arXiv:1902.10503 [astro-ph.CO] .
  * Heckman  _et al._ [2019a] J. J. Heckman, C. Lawrie, L. Lin, and G. Zoccarato, Fortsch. Phys. 67, 1900057 (2019a), arXiv:1811.01959 [hep-th] .
  * Heckman  _et al._ [2019b] J. J. Heckman, C. Lawrie, L. Lin, J. Sakstein, and G. Zoccarato, Fortsch. Phys. 67, 1900071 (2019b), arXiv:1901.10489 [hep-th] .
  * Heckman  _et al._ [2022] J. J. Heckman, A. Joyce, J. Sakstein, and M. Trodden, Int. J. Mod. Phys. A 37, 2250201 (2022), arXiv:2208.02267 [hep-th] .
  * Hinterbichler and Khoury [2010] K. Hinterbichler and J. Khoury, Phys. Rev. Lett. 104, 231301 (2010), arXiv:1001.4525 [hep-th] .
  * Hinterbichler  _et al._ [2011] K. Hinterbichler, J. Khoury, A. Levy, and A. Matas, Phys. Rev. D 84, 103521 (2011), arXiv:1107.2112 [astro-ph.CO] .
  * Fardon  _et al._ [2004] R. Fardon, A. E. Nelson, and N. Weiner, JCAP 10, 005 (2004), arXiv:astro-ph/0309800 .
  * Brookfield  _et al._ [2006] A. W. Brookfield, C. van de Bruck, D. F. Mota, and D. Tocchini-Valentini, Phys. Rev. D 73, 083515 (2006), [Erratum: Phys.Rev.D 76, 049901 (2007)], arXiv:astro-ph/0512367 .
  * Sakstein and Trodden [2020] J. Sakstein and M. Trodden, Phys. Rev. Lett. 124, 161301 (2020), arXiv:1911.11760 [astro-ph.CO] .
  * Carrillo González  _et al._ [2021] M. Carrillo González, Q. Liang, J. Sakstein, and M. Trodden, JCAP 04, 063 (2021), arXiv:2011.09895 [astro-ph.CO] .
  * Bhattacharya  _et al._ [2024] S. Bhattacharya, G. Borghetto, A. Malhotra, S. Parameswaran, G. Tasinato, and I. Zavala, (2024), arXiv:2405.17396 [astro-ph.CO] .


[a,b]Eric B. Gregory

#  Exotic states in the $\pi D$ system

Feng-Kun Guo  Christoph Hanhart  Stefan Krieg  Thomas Luu

###### Abstract

In this proceedings we consider several states, namely the $D^{*}_{s0}(2317)$,
$D_{s1}(2460)$, $D^{*}_{0}(2300)$ and $D_{1}(2430)$, which appear to defy
description as simple quark-antiquark pairs. Theoretical input from unitarized
chiral perturbation theory suggests they can be understood as emerging from
Goldstone-Boson–$D$-meson scattering.

We present results from an $SU(3)$ flavor-symmetric lattice QCD simulation at
large pion masses suggesting that there exists a $\pi D$ bound state in the
flavor-sextet representation that cannot emerge for quark-antiquark states,
but that appears naturally from the multiquark states. Moreover, we find
repulsion in the [15] representation, which establishes the pattern predicted
for the interactions of Goldstone bosons with $D$ mesons. This suggests these
states may have the structure of hadronic molecules.

##  1 Introduction

A major goal of modern particle physics is to explain the nature of particles
observed in experiments. We focus on several observed particles, namely
$D^{*}_{s0}(2317)$, $D_{s1}(2460)$, $D^{*}_{0}(2300)$ and $D_{1}(2430)$, which
defy explanation through the quark model as simple $q\overline{q}$ meson
states. The $D^{*}_{s0}(2317)$ and $D_{s1}(2460)$ states are lighter than
quark model predictions would suggest [1, 2, 3]. Furthermore, we expect that
non-strange states should be about 150 MeV lighter than their strange
counterparts. However, $D^{*}_{0}(2300)$ and $D_{1}(2430)$ are similar in mass
to their corresponding strange states.

The likely explanation is that these states are exotic. Insight into their
nature has come from unitarized chiral perturbation theory (U$\chi$PT) [4, 5,
6, 7, 8]. U$\chi$PT suggests that $D^{*}_{s0}(2317)$ is a $DK$ molecule and
that the very broad $D^{*}_{0}(2300)$ state contains two poles around 2105 MeV
and 2451 MeV. Also the $D^{*}_{s0}(2317)$ is the $SU(3)$-flavor partner of the
lower of these two poles, while the upper pole is a member of an
$SU(3)$-sextet.

Several lattice studies have added information about these states, with [9]
examining $DK$ scattering. The Regensburg group calculated $D^{*}_{s0}(2317)$
and $D_{s1}(2460)$ masses [10]. The Hadron Spectrum Collaboration predicted
that the lightest $D^{*}_{0}$ should be below 2300MeV. For reviews, see Refs.
[11, 12].

There remain, however, questions regarding the structure of these states: are
they compact tetraquarks of the diquark-antidiquark type or are they hadronic
molecules, as suggested by U$\chi$PT?

Further insight comes from understanding the nature of the $SU(3)$-flavor
multiplets. In a flavor-symmetric world, a four-quark state
$c\overline{q}q\overline{q}$ breaks down as

$[\bar{3}]\otimes[8]=[15]\oplus[6]\oplus[\bar{3}]\ ,$ (1)

ignoring the $[\overline{3}]$ arising from $c\overline{q}$ coupling to the
flavor-singlet $q\overline{q}$. Dmitrasinovic predicted that in a tetraquark
configuration, all multiplets are attractive [13]. On the other hand,
U$\chi$PT predicts that for hadronic molecules, the $[\overline{3}]$ would be
the most attractive multiplet, followed by the $[6]$, with the $[15]$ being
repulsive [4, 14, 15].

With these conflicting predictions, determination of the attractive or
repulsive nature of the $SU(3)$ multiplets could lend strong support to either
the compact tetraquark or hardonic molecule hypothesis. With this in mind, we
set out to do a $SU(3)$-flavor symmetric lattice calculation of the $D\pi$
system.

##  2 Lattice Calculation

###  2.1 Ensemble tuning

The U$\chi$PT calculation of Du et al. [6] suggests that $SU(3)$ flavor
symmetric light quarks tuned so that the pion mass $M_{\pi}$ roughly in the
range 600 – 700 MeV should produce a near-threshold virtual or bound state in
the$[6]$ representation. Our goal then was to generate ensembles with three
light quarks tuned to give the flavor symmetric pions in this mass range, and
the charm quark at its physical mass.

Using CHROMA [16], with either QPHIX[18] or QUDA[17], we generated 63 tuning
ensembles with clover Wilson fermions with six iterations of stout smearing.
We initially explored three values of the coupling, $\beta=3.4,3.5$ and $3.6$.

The tuning procedure is complicated by the lack of light quarks near the
physical mass range. This prevents us from setting the lattice scale with
traditional light hadron observables and methods such as renormalization flow,
which are calibrated with the light hadron spectrum. Instead we use a three-
step tuning procedure and rely on charmonium splitting to set the lattice
scale.

In the first step we construct the ratio

$R=\frac{M_{J/\psi}-M_{\eta_{c}}}{M_{J/\psi}}$ (2)

and vary the bare charm quark mass until the physical value of $R=0.365$ is
achieved. We then read off the value of $am_{c}$ that gives physical charm
quarks. This step is illustrated in Fig. 1.

Figure 1: Tuning $m_{c}$. For each $\beta$, we find the value of $am_{c}$
where the linear fit to measured values of the charmonium splitting ratio
(dashed lines) crosses the physical value $R=0.0365$ (blue line).

Once we have determined the target $am_{c}$ for each $\beta$, we use the value
of the charmonium splitting $aM_{J/\psi}-aM_{\eta_{c}}$ to establish the
lattice spacing:

$a=\frac{\left(aM_{J/\psi}-aM_{\eta_{c}}\right)_{\rm
latt}}{\left(M_{J/\psi}-M_{\eta_{c}}\right)_{\rm
phys}}=\frac{\left(aM_{J/\psi}-aM_{\eta_{c}}\right)_{\rm latt}}{113~{}{\rm
MeV}}.$ (3)

This process is illustrated in Fig. 2.

Figure 2: Determining the lattice scale $a$. Where the linear fits to lattice
values of the splitting $a(M_{J/\psi}-M_{\eta_{c}})$ intersect the value of
$am_{c}$ determined for each $\beta$ in the first step, we use this splitting
to determine the lattice spacing $a$ in GeV-1 (horizontal dashed lines to the
right axis).

We note that we do not include disconnected diagrams in our calculation of
$M_{J/\psi}$ and $M_{\eta_{c}}$ which we estimate introduces a 10% systematic
overestimation of $\left(aM_{J/\psi}-aM_{\eta_{c}}\right)_{\rm latt}$ and
hence a corresponding error in $a$ [19].

The final step is to find the value of the bare light quark mass, $am_{s}$,
that produces pions in the 600 – 700 MeV range. This is straightforward now
that we have an estimate for the lattice spacing $a$. See Fig. 3.

Figure 3: Tuning the light ($SU(3)$) quark masses. The dashed diagonal lines
are linear fits to ($m_{q}$,$aM_{\pi}$). The dashed vertical lines drop from
the point the fit lines pass through $a(650~{}{\rm MeV})$. The vertical solid
lines correspond to 600 MeV and 700 MeV. Different $a$ for each $\beta$ is
implied.

We reiterate that, with the exception of the charm quark, we are not tuning to
physical quark masses, nor are we attempting precision spectroscopy requiring
a continuum extrapolation. For our purposes a single lattice spacing is
sufficient to investigate the nature of the $SU(3)$ multiplets. For this
reason, we focus from here on the $\beta=3.6$ ensembles.

We find that for $\beta=3.6$, $a=0.27(2)_{\rm stat}(2)_{\rm sys}$ GeV-1. At
this lattice spacing, our ensemble at $am_{q}=-0.013$, $am_{c}=0.25$ is very
close to the target point and corresponds to $M_{\pi}=612(90)$ MeV (with the
uncertainty dominated by the lattice spacing determination). At this target
point we generated three ensembles, each with $L_{t}=64$ lattice units in the
time direction, but varying spatial volumes of $L_{s}^{3}=32^{3},40^{3}$ and
$48^{3}$. These correspond roughly to spatial sizes of 1.6, 2.1 and 2.6 fm,
respectively. For each we generated 2500 – 2700 trajectories.

###  2.2 Interpolating operator design and measurement

Because the $[\overline{3}]$ is not useful for distinguishing between
tetraquarks and hadronic molecules, and because of the additional
computational difficulty in resolving its disconnected diagrams, we do not
address the $[\overline{3}]$ irrep in the lattice simulation described below.
We focus instead on the $[6]$ and $[15]$.

To construct the appropriate interpolating operators for the $[6]$ and $[15]$
we first construct the needed $SU(3)$ flavor states within a tensor basis
[20]. Since the $c$ quark is in an $SU(3)$ singlet, the states for the
remaining degenerate light quarks can be constructed with appropriate
projection onto the $[6]$ and $[15]$ irreps. Table 1 shows the states and
their quantum numbers for the $[6]$ irrep.

Table 1: The light quark content of the states in the $[6]$ representation and their associated quantum numbers. $T^{2}_{a}$ is the Casimir operator, $I_{z}$ is the third component of isospin, and $Y$ is the hypercharge. state | components | $T_{a}^{2}$ | $I_{z}$ | $Y$
---|---|---|---|---
1 | $-|u\bar{u}\bar{d}\rangle\frac{1}{2}+|u\bar{d}\bar{u}\rangle\frac{1}{2}-|s\bar{d}\bar{s}\rangle\frac{1}{2}+|s\bar{s}\bar{d}\rangle\frac{1}{2}$ | $\frac{10}{3}$ | $+\frac{1}{2}$ | $-\frac{1}{3}$
2 | $|d\bar{u}\bar{d}\rangle\frac{1}{2}-|d\bar{d}\bar{u}\rangle\frac{1}{2}-|s\bar{u}\bar{s}\rangle\frac{1}{2}+|s\bar{s}\bar{u}\rangle\frac{1}{2}$ | $\frac{10}{3}$ | $-\frac{1}{2}$ | $-\frac{1}{3}$
3 | $|u\bar{u}\bar{s}\rangle\frac{1}{2}-|u\bar{s}\bar{u}\rangle\frac{1}{2}-|d\bar{d}\bar{s}\rangle\frac{1}{2}+|d\bar{s}\bar{d}\rangle\frac{1}{2}$ | $\frac{10}{3}$ | 0 | $+\frac{2}{3}$
4 | $|d\bar{s}\bar{u}\rangle\frac{1}{\sqrt{2}}-|d\bar{u}\bar{s}\rangle\frac{1}{\sqrt{2}}$ | $\frac{10}{3}$ | $-1$ | $+\frac{2}{3}$
5 | $|u\bar{s}\bar{d}\rangle\frac{1}{\sqrt{2}}-|u\bar{d}\bar{s}\rangle\frac{1}{\sqrt{2}}$ | $\frac{10}{3}$ | $+1$ | $+\frac{2}{3}$
6 | $|s\bar{d}\bar{u}\rangle\frac{1}{\sqrt{2}}-|s\bar{u}\bar{d}\rangle\frac{1}{\sqrt{2}}$ | $\frac{10}{3}$ | 0 | $-\frac{4}{3}$

If we choose state 5, we would write down the operator:

$O^{5}_{[6]}(x^{\prime};x)=\frac{1}{\sqrt{2}}\left\\{\left[\bar{s}(x^{\prime})\Gamma
c(x^{\prime})\right]\left[\bar{d}(x)\Gamma
u(x)\right]-\left[\bar{d}(x^{\prime})\Gamma
c(x^{\prime})\right]\left[\bar{s}(x)\Gamma u(x)\right]\right\\}.$ (4)

We could instead write the operator for state 6:

$O^{5}_{[6]}(x^{\prime};x)=\frac{1}{\sqrt{2}}\left\\{\left[\bar{s}(x^{\prime})\Gamma
c(x^{\prime})\right]\left[\bar{d}(x)\Gamma
u(x)\right]-\left[\bar{d}(x^{\prime})\Gamma
c(x^{\prime})\right]\left[\bar{s}(x)\Gamma u(x)\right]\right\\}.$ (5)

Regardless, at the SU(3) point we get the same expression when we complete the
contractions to obtain the correlator:

$\langle
O^{i}_{[6]}(y^{\prime};y)\bar{O}^{i}_{[6]}(x;x)\rangle=\text{Tr}\left[\Gamma\gamma_{5}\mathcal{S}^{\dagger}_{y^{\prime};x}\gamma_{5}\Gamma\mathcal{S}_{y^{\prime};x}\right]\text{Tr}\left[\Gamma\gamma_{5}\mathcal{S}^{\dagger}_{y;x}\gamma_{5}\Gamma\mathcal{C}_{y;x}\right]\\\
+\text{Tr}\left[\Gamma\gamma_{5}\mathcal{S}^{\dagger}_{y;x}\gamma_{5}\Gamma\mathcal{S}_{y^{\prime};x}\Gamma\gamma_{5}\mathcal{S}^{\dagger}_{y^{\prime};x}\gamma_{5}\Gamma\mathcal{C}_{y;x}\right]\
,$ (6)

The contraction for the $[15]$, $\langle
O^{i}_{[d]}(y^{\prime};y)\bar{O}^{i}_{[d]}(x;x)\rangle^{[15]}$ is identical to
Eq. (6), but with a relative minus sign between the terms. In our calculations
we consider only $\Gamma=\gamma_{5}$.

###  2.3 Analysis

Figure 4: Eight hadron correlators used for simultaneous 2-state fits with
$t_{\rm min}=5$ (see text). From top to bottom, the correlators represent:
$\pi_{\rm pt}$, $\pi_{\rm sm}$, $D_{\rm pt}$, $D_{\rm sm}$, $[6]_{\rm pt}$,
$[15]_{\rm pt}$, $[6]_{\rm sm}$, and $[15]_{\rm sm}$, with the subscripts “pt"
and “sm" referring to point and smeared sink operators. Lattice data error
bars are too small to be seen.

We measure correlators for the $\pi$, $D$, as well as the $[6]$ and $[15]$
states, using point and smeared sink operators for each. This gives us eight
different correlators, which we fit simultaneously using

$C_{P,s}(t)=\sum_{j=0}^{(N-1)}A_{P,s,j}\cosh\left(M_{P,j}(t-L_{t}/2)\right)$
(7)

for $P=\\{\pi,D\\}$ and

$C_{P,s}(t)=B_{s}\cosh\left((M_{D}-M_{\pi})(t-L_{t}/2)\right)+A_{P,s,0}\cosh\left((\Delta_{M_{P}}+M_{D,0}+M_{\pi,0})(t-L_{t}/2)\right)\\\
+\sum_{j=1}^{(N-1)}A_{P,s,j}\cosh\left(M_{p,j}(t-L_{t}/2)\right),$ (8)

for $p=\\{[6],[15]\\}$, and $s=${point, smeared} sink operators. The overall
fit form is

${\mathcal{C}}(t,P,s)=\delta_{P,P^{\prime}}\delta_{s,s^{\prime}}C_{P^{\prime},s^{\prime}}(t).$
(9)

The first term in Eq. (8) is due to forward-propagating $\pi$ and backwards-
propagating $D$ and vice versa. The parameters of interest are

$\displaystyle\Delta M_{[6]}$ $\displaystyle\equiv$ $\displaystyle
M_{[6]}-\left(M_{D}+M_{\pi}\right),$ $\displaystyle\Delta M_{[15]}$
$\displaystyle\equiv$ $\displaystyle M_{[15]}-\left(M_{D}+M_{\pi}\right).$
(10)

These will be negative for attractive states and positive for repulsive
scattering states. The correlators and sample fits are shown in Fig. 4.

We performed ground-state ($N=1$) and ground-state plus first-excited-state
($N=2$) fits, varying a symmetric fit range $t_{\rm min}...(L_{t}-t_{\rm
min})$. We used jackknife resampling with a bin size of 15 (150 trajectories)
to extract uncertainties on the fit parameters.

##  3 Results, conclusions and future work

Figure 5 shows the resulting $\Delta_{M}$ values. We find a statistically
significant negative signal for $\Delta_{M_{[6]}}$, with little variation
across the three volumes. For the $[15]$ we see $\Delta_{M_{[15]}}>0$ for all
three volumes. However, $\Delta_{M_{[15]}}$ seems to approach zero as the
volume increases, as would be expected in a scattering state.

Figure 5: Mass shifts $\Delta{M_{[6]}}$ and $\Delta{M_{[15]}}$ from two-state
fits ($N=2$) as a function of $t_{\rm min}$. A visible symbol indicates that
the fit confidence ($p$-value) is near 1. Error bars are larger on the
$40^{3}$ fits because there are fewer measurements per configuration.

Taken together these are consistent with the U$\chi$PT predictions of the
$SU(3)$ $[6]$ and $[15]$ multiplets, and thus support the interpretation of
the lowest-lying positive parity charmed mesons as hadronic molecules. In
particular, the [6] state is beyond the simple $c\bar{q}$ configurations and
is explicitly exotic.

Our next step in future work is to remove the remaining ambiguity about these
results by increasing the statistics, using more advanced fitting techniques,
and performing a Lüscher finite-volume analysis.

##  4 Acknowledgements

We thank Alessandro Pilloni, Evan Berkowitz, Andrea Shindler, David Wilson and
Christopher Thomas for useful discussions. We are is indebted to André Walker-
Loud for his insightful discussions related to backward propagating states. We
are grateful to Balint Joo for assistance with CHROMA and to Kate Clark and
Mathias Wagner for guidance with QUDA. This work is supported in part by the
National Natural Science Foundation of China (NSFC) under Grant No. 11835015,
No. 12047503, and No. 11961141012, by the NSFC and the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) through the funds
provided to the Sino-German Collaborative Research Center “Symmetries and the
Emergence of Structure in QCD” (NSFC Grant No. 12070131001, DFG Project-ID
196253076 – TRR110), and by the Chinese Academy of Sciences (CAS) under Grant
No. XDB34030000 and No. QYZDB-SSW-SYS013.

## References

  * [1] S. Godfrey and N. Isgur, Phys. Rev. D 32, 189-231 (1985) doi:10.1103/PhysRevD.32.189
  * [2] S. Godfrey and K. Moats, Phys. Rev. D 93, no.3, 034035 (2016) doi:10.1103/PhysRevD.93.034035 [arXiv:1510.08305 [hep-ph]].
  * [3] D. Ebert, R. N. Faustov and V. O. Galkin, Eur. Phys. J. C 66, 197-206 (2010) doi:10.1140/epjc/s10052-010-1233-6 [arXiv:0910.5612 [hep-ph]].
  * [4] E. E. Kolomeitsev and M. F. M. Lutz, Phys. Lett. B 582, 39-48 (2004) doi:10.1016/j.physletb.2003.10.118 [arXiv:hep-ph/0307133 [hep-ph]].
  * [5] M. Albaladejo, P. Fernandez-Soler, F.-K. Guo and J. Nieves, Phys. Lett. B 767, 465-469 (2017) doi:10.1016/j.physletb.2017.02.036 [arXiv:1610.06727 [hep-ph]].
  * [6] M.-L. Du, M. Albaladejo, P. Fernández-Soler, F.-K. Guo, C. Hanhart, U.-G. Meißner, J. Nieves and D.-L. Yao, Phys. Rev. D 98, no.9, 094018 (2018) doi:10.1103/PhysRevD.98.094018 [arXiv:1712.07957 [hep-ph]].
  * [7] L. Liu, K. Orginos, F.-K. Guo, C. Hanhart and U.-G. Meißner, Phys. Rev. D 87, no.1, 014508 (2013) doi:10.1103/PhysRevD.87.014508 [arXiv:1208.4535 [hep-lat]].
  * [8] X. Y. Guo, Y. Heo and M. F. M. Lutz, Phys. Rev. D 98, no.1, 014510 (2018) doi:10.1103/PhysRevD.98.014510 [arXiv:1801.10122 [hep-lat]].
  * [9] D. Mohler, C. B. Lang, L. Leskovec, S. Prelovsek and R. M. Woloshyn, Phys. Rev. Lett. 111, no.22, 222001 (2013) doi:10.1103/PhysRevLett.111.222001 [arXiv:1308.3175 [hep-lat]].
  * [10] G. S. Bali, S. Collins, A. Cox and A. Schäfer, Phys. Rev. D 96, no.7, 074501 (2017) doi:10.1103/PhysRevD.96.074501 [arXiv:1706.01247 [hep-lat]].
  * [11] F.-K. Guo, C. Hanhart, U.-G. Meißner, Q. Wang, Q. Zhao and B.-S. Zou, Rev. Mod. Phys. 90, no.1, 015004 (2018) doi:10.1103/RevModPhys.90.015004 [arXiv:1705.00141 [hep-ph]].
  * [12] F.-K. Guo, EPJ Web Conf. 202, 02001 (2019) doi:10.1051/epjconf/201920202001
  * [13] V. Dmitrasinovic, Phys. Rev. Lett. 94, 162002 (2005) doi:10.1103/PhysRevLett.94.162002
  * [14] F.-K. Guo, P.-N. Shen, H.-C. Chiang, R.-G. Ping and B.-S. Zou, Phys. Lett. B 641, 278-285 (2006) doi:10.1016/j.physletb.2006.08.064 [arXiv:hep-ph/0603072 [hep-ph]].
  * [15] F.-K. Guo, C. Hanhart and U.-G. Meißner, Eur. Phys. J. A 40, 171-179 (2009) doi:10.1140/epja/i2009-10762-1 [arXiv:0901.1597 [hep-ph]].
  * [16] R. G. Edwards (LHPC Collaboration), B. Joó (UKQCD Collaboration), "The Chroma Software System for Lattice QCD", arXiv:hep-lat/0409003, Proceedings of the 22nd International Symposium for Lattice Field Theory (Lattice2004), Nucl. Phys B1 40 (Proc. Suppl) p832, 2005.
  * [17] M. A. Clark, R. Babich, K. Barros, R. C. Brower, C. Rebbi, Solving Lattice QCD systems of equations using mixed precision solvers on GPUs arXiv:0911.3191v1 [hep-lat]
  * [18] https://github.com/JeffersonLab/qphix
  * [19] D. Hatton et al. [HPQCD], Phys. Rev. D 102, no.5, 054511 (2020) doi:10.1103/PhysRevD.102.054511 [arXiv:2005.01845 [hep-lat]].
  * [20] H. Georgi, Front. Phys. 54, 1-255 (1982)