# A Unified Taxonomy-Guided Instruction Tuning Framework for
Entity Set Expansion and Taxonomy Expansion

Yanzhan Shen, Yu Zhang11footnotemark: 1, Yunyi Zhang, Jiawei Han
University of Illinois Urbana-Champaign, IL, USA
{yanzhen4, yuz9, yzhan238<EMAIL_ADDRESS>Equal contribution

###### Abstract

Entity Set Expansion, Taxonomy Expansion, and Seed-Guided Taxonomy
Construction are three representative tasks that can be used to automatically
populate an existing taxonomy with new entities. However, previous approaches
often address these tasks separately with heterogeneous techniques, lacking a
unified perspective. To tackle this issue, in this paper, we identify the
common key skills needed for these tasks from the view of taxonomy
structures—finding “siblings” and finding “parents”—and propose a unified
taxonomy-guided instruction tuning framework to jointly solve the three tasks.
To be specific, by leveraging the existing taxonomy as a rich source of entity
relationships, we utilize instruction tuning to fine-tune a large language
model to generate parent and sibling entities. Extensive experiments on
multiple benchmark datasets demonstrate the effectiveness of TaxoInstruct,
which outperforms task-specific baselines across all three tasks.111Code can
be found at https://github.com/yanzhen4/TaxoInstruct

A Unified Taxonomy-Guided Instruction Tuning Framework for
Entity Set Expansion and Taxonomy Expansion

Yanzhan Shen††thanks: Equal contribution, Yu Zhang11footnotemark: 1, Yunyi
Zhang, Jiawei Han University of Illinois Urbana-Champaign, IL, USA {yanzhen4,
yuz9, yzhan238<EMAIL_ADDRESS>

##  1 Introduction

Entities play a fundamental role in natural language understanding and can
benefit various downstream tasks such as semantic search Shen et al. (2018b),
question answering Ma et al. (2023), and text generation Wang et al. (2021).
To better describe the semantics of entities, taxonomies are usually
constructed in different domains, such as science Shen et al. (2018c); Coletti
and Bleich (2001), e-commerce Mao et al. (2020), and social media Gonçalves et
al. (2019), to characterize the parent-child relationship between entities. In
many cases, taxonomies are initially curated by domain experts. However,
because of the constant and rapid emergence of novel concepts, automatically
enriching a taxonomy with new entities becomes necessary to ensure its
freshness and completeness.

Previous studies have considered three representative tasks for entity
enrichment. (1) Entity Set Expansion Wang and Cohen (2007); Rong et al.
(2016); Shen et al. (2017): Given a set of entities belonging to a certain
semantic class, find new entities also in that class. For example, given the
seeds $\\{$natural language processing, computer vision, data mining$\\}$, an
entity set expansion algorithm should return other areas such as cybersecurity
and human-computer interaction. From the taxonomy perspective, this task can
be viewed as finding new “siblings” of existing entities. (2) Taxonomy
Expansion Shen et al. (2020); Yu et al. (2020); Jiang et al. (2022): Insert a
provided new entity into an existing taxonomy by finding its most suitable
“parent”. For instance, if the existing taxonomy has root scientific
disciplines, with children entities computer science, chemistry, biology, and
physics, given the new concept natural language processing, a Taxonomy
Expansion model should put it as a child of computer science. (3) Seed-Guided
Taxonomy Construction Shen et al. (2018a): First discover new entities related
to a given seed taxonomy and then insert them into the taxonomy. For example,
given the seeds $\\{$computer science, chemistry, computer vision, organic
chemistry$\\}$ and their parent-child relationship (i.e., compute science
$\leftarrow$ computer vision, chemistry $\leftarrow$ organic chemistry), a
Seed-Guided Taxonomy Construction algorithm needs to find new entities at the
granularity of chemistry and computer science (e.g., physics and biology),
find new entities at the granularity of computer vision and organic chemistry
(e.g., database system and materials chemistry), and figure out the parent-
child relationship between expanded entities (e.g., database system
$\rightarrow$ computer science, materials chemistry $\rightarrow$ chemistry).
This can be viewed as pipelining the steps of finding “siblings” and finding
“parents”.

Figure 1: Illustrations of the three tasks.

Although the three aforementioned tasks have been extensively explored,
existing approaches do not consider them in a unified way. Each previous study
focuses on only one of these tasks with less concern for their commonality;
the best performance in different tasks is achieved by different models with
task-specific techniques. However, as we have pointed out, all three tasks can
be cast as finding entities that have a specific type of relationship with the
given entities: Entity Set Expansion is essentially finding “siblings”;
Taxonomy Expansion relies on finding “parents”; Seed-Guided Taxonomy
Construction is a combination of both. Therefore, a model that is trained to
predict siblings and parents can be directly applied to all the tasks.

Contributions. Inspired by the idea above, in this paper, we propose
TaxoInstruct, a unified taxonomy-guided instruction tuning framework for
Entity Set Expansion, Taxonomy Expansion, and Seed-Guided Taxonomy
Construction. We seek supervision from the existing taxonomy, which serves as
a rich source of sibling-sibling and parent-child relationships between
entities. Then, we propose to utilize instruction tuning Wei et al. (2022) to
fine-tune a large language model (LLM) Touvron et al. (2023) for generating
parent/sibling entities. We conduct comprehensive experiments to examine the
effectiveness of TaxoInstruct. Empirical results demonstrate that
TaxoInstruct, as a unified framework, significantly outperforms task-specific
baselines in the corresponding tasks.

To summarize, this study makes three major contributions: (1) We propose a
unified solution to the tasks of Entity Set Expansion, Taxonomy Expansion, and
Seed-Guided Taxonomy construction, which are separately studied in previous
work. (2) We present a taxonomy-guided instruction tuning approach that seeks
supervision from the existing taxonomy to find parent and sibling entities.
(3) We validate the efficacy of our framework through extensive performance
comparisons, hyperparameter analyses, and case studies.

##  2 Task Definition

Entity Set Expansion. As shown in Figure 1 a), given a few example entities
(also known as “seeds”), the entity set expansion task Wang and Cohen (2007);
Rong et al. (2016); Shen et al. (2017) aims to find a set of sibling entities
that belong to the same semantic class as the seeds. Formally, we have the
following task definition.

######  Definition 1.

(Entity Set Expansion) Given a small set of seed entities
$\mathcal{S}=\\{s_{1},s_{2},...,s_{M}\\}$, the task is to discover more
entities $\mathcal{S}^{+}=\\{s_{M+1},s_{M+2},$ $...,s_{M+N}\\}$, where
$s_{1},s_{2},...,s_{M+N}$ fall into the same semantic category.

Taxonomy Expansion. As shown in Figure 1 b), given an existing taxonomy and a
set of new entities, Taxonomy Expansion Shen et al. (2020); Yu et al. (2020)
aims at inserting the new entities into the taxonomy. This process is
facilitated by finding a proper parent node in the existing taxonomy for each
new entity. Formally,

######  Definition 2.

(Taxonomy Expansion) Given an existing taxonomy $\mathcal{T}$ (which contains
a set of entities $\mathcal{S}$ and the parent-child relationship between the
entities
$\textsc{Parent}(\cdot):\mathcal{S}\rightarrow\mathcal{S}\cup\\{\textrm{ROOT}\\}$)
and a set of new entities $\mathcal{S}^{+}$, the task is to expand the
taxonomy to a more complete one $\mathcal{T}^{+}$ with entities
$\mathcal{S}\cup\mathcal{S}^{+}$ and the parent-child relationship
$\textsc{Parent}^{+}(\cdot):\mathcal{S}\cup\mathcal{S}^{+}\rightarrow\mathcal{S}\cup\\{\textrm{ROOT}\\}$.

Figure 2: Illustration of TaxoInstruct instructions.

Seed-Guided Taxonomy Construction. As shown in Figure 1 c), Seed-Guided
Taxonomy Construction Shen et al. (2018a) deals with the case where we need to
first find a set of new entities to be inserted to the taxonomy and then find
the proper parent node for each new entity.

######  Definition 3.

(Seed-Guided Taxonomy Construction) Given an existing taxonomy $\mathcal{T}$,
the task is to find a set of entities $\mathcal{S}^{+}$ that share similar
topic as the taxonomy $\mathcal{T}$, and then expand the taxonomy to a more
complete taxonomy $\mathcal{T}^{+}$ with entities
$\mathcal{S}\cup\mathcal{S}^{+}$ and the parent-child relationship
$\textsc{Parent}^{+}(\cdot):\mathcal{S}\cup\mathcal{S}^{+}\rightarrow\mathcal{S}\cup\\{\textrm{ROOT}\\}$.

##  3 Model

We introduce our unified taxonomy-guided instruction tuning framework,
TaxoInstruct, in this section. First, we present how we use taxonomy-guided
instruction tuning on each of the tasks respectively in Section 3.2, 3.1 and
3.3. Then, in Section 3.4, we propose a unified taxonomy-guided tuning
framework which trains the LLM to understand taxonomic relations.

###  3.1 Instruction Tuning for Entity Set Expansion

Given a set of seed entities $\mathcal{S}=\\{s_{1},s_{2},...,s_{n}\\}$
belonging to the same semantic class, the Entity Set Expansion task aims to
find other entities belonging to the same class. It thus enforces two
restrictions on the expanded entities. First, the expanded entities can be
classified into the same category as the seed entities. For example, both
“heart diseases” and “brain stem stroke” can be classified into the category
“diseases”. Second, they must also share the same granularity. For example,
the entity “heart disease” is at the same level as “vascular disease”, but not
as “brain stem stroke” which is of finer granularity. These two restrictions
are inherently describing the concept of “siblings” in a taxonomy, since
“siblings” share a single parent (being classified into the same category) and
reside in the same level (having the same granularity).

Therefore, we formulate the Entity Set Expansion task under the taxonomy
setting as finding other siblings of the seed entities. Utilizing LLMs’ power
of following instructions Wei et al. (2022), we formulate the task as follows:
given a set of seed entities $\\{s_{1},s_{2},...s_{n}\\}$ as the Query that
share the same parent node $s_{parent}$, we Instruct an chat-completion LLM
(e.g., LLaMA-7B) Touvron et al. (2023) to find more children of this parent
node, and the model should Output siblings of the seed entities, which are
denoted as $\\{\hat{s}_{1},\dots,\hat{s}_{m}\\}$.

However, the parent entity $s_{parent}$ is not available in the standard
Entity Set Expansion task. Thus, we first prompt the LLM to generate the
parent entity for the seed set $\mathcal{S}$. Following the same (Instruction,
Query, Output) schema, we construct instructions as follows:

> Instruction: Given a list of entities, output the most likely parent class
> for the entity given by user.
>
> Query: Find the parent for $s_{1}$, $s_{2}$, … $s_{n}$.
>
> Output: The parent class is

The generated parent entity, $s_{parent}$, can be used to guide the expansion
process. Because the Entity Set Expansion task normally contains a very small
number of seed entities (e.g., only 3 seeds), it is hard to get sufficient
self-supervision data from the seeds. Therefore, we do not fine-tune the LLM
on the task specific data and directly use it to expand the seed set. We
construct instructions as follows:

> Instruction: Given a category and an entity set belonging to this category,
> output other entities of this category that share the same granularity as
> the seeds.
>
> Query: Find other entities belonging to category $s_{parent}$ and sharing
> the same granularity as the seeds $s_{1}$, $s_{2}$, … $s_{n}$.
>
> Output: The expanded entities are

We feed the instructions into the LLM to generate a set of expanded entities,
$\\{\hat{s}_{1},\dots,\hat{s}_{m}\\}$

Empirically, we find that shuffling the seed entities in the query will lead
to different outputs by the LLM due to the sequential nature of language
modeling.

Therefore, If we are given $n$ seed entities, we generate all their
permutations $Perm(S)$, each of which will be used to construct a tuple of
(Instruction, Query). By feeding them individually into the LLM, we can get
$n!$ sets of expanded entities:
$[\mathcal{S}^{1},\mathcal{S}^{2},\dots,\mathcal{S}^{n!}]$. We take the union
of them to get a single set of unique entities
$\mathcal{S}^{output}=\bigcup_{j=1}^{n!}\mathcal{S}^{j}$.

Because in previous works, the Entity Set Expansion task is formulated to
generate a ranked list of entities, here we perform an additional step to rank
the entities generated by the LLM. We use an moderate-size auxiliary PLM
(e.g., BERT Devlin et al. (2019)) to compute the similarity score of each
entity $s_{p}\in\mathcal{S}^{output}$ with respect to $s_{parent}$:

$score(s_{p},s_{parent})=\operatorname{sim}\left(h(s_{p}),h(s_{parent})\right),$
(1)

where $sim(\cdot,\cdot)$ is the cosine similarity, and $h(\cdot)$ is the
auxiliary PLM that maps an entity to its embedding. By ranking the similarity
scores, we can then get a ranked list, $\mathcal{S}^{output}_{sorted}$, of the
unique entities generated by the LLM.

###  3.2 Instruction Tuning for Taxonomy Expansion

Given a query entity, the goal of Taxonomy Expansion is to select its parent
node from the taxonomy. In this work, we Instruct an LLM to select the parent
node from a provided list of candidates given a Query entity, and the model
should Output its correct parent node.

To help the LLM better understand parent-child relationship and domain
knowledge, we first fine-tune it on self-supervision data that is
automatically constructed from the provided taxonomy.

Following a similar schema as Entity Set Expansion, we construct tuples of
(Instruction, Query, Output). They will be used to fine-tune an LLM to find
the correct parent node for the query entity. We include a set of candidate
parents in the Instruction, and the LLM should choose the right parent from
these candidates.

However, if we use all the nodes in the taxonomy as candidates and incorporate
them into the instruction, the LLM may be overwhelmed by the overly large
label space and can hardly follow the instruction to perform the task.
Besides, the taxonomy may contain thousands of nodes and can make the
instruction exceed the input length limit of the LLM. The LLM will
consequently fail to choose the correct parent from the candidates.

Therefore, we propose to use an auxiliary PLM to first retrieve a set of
candidate nodes from the taxonomy and thus reduce the label space for the LLM.
More specifically, given a node in $s_{i}\in\mathcal{S}$, we will select
top-$k$ (e.g., $k=20$) entities $T^{i}_{k}\subseteq S$ with the highest
similarity to $s_{i}$.

$T^{i}_{k}=\operatorname*{arg\,max}_{T\subseteq S,|T|=k}\sum_{s_{j}\in
T}sim\left(h(s_{i}),h(s_{j})\right),$ (2)

where $sim(\cdot,\cdot)$ is the cosine similarity and $h(\cdot)$ is the
auxiliary PLM similar to the previous section.

We utilize the existing parent-child relation in the existing taxonomy as
self-supervision data to fine-tune the LLM. For a node $s_{i}$ and its parent
$s^{i}_{parent}$ in the taxonomy, we first use eq. 2 to find the set of
top-$k$ nodes that are most similar to $s_{i}$,
$T^{i}_{k}=\\{\hat{s}^{i}_{1},\dots,\hat{s}^{i}_{k}\\}$. Then, the candidate
parents for $s_{i}$ is $Cand^{i}_{k}=T^{i}_{k}\cup\\{s^{i}_{parent}\\}$, and
we fine-tune the LLM to identify the true parent from the candidate list.
Because the candidates will be sequential in the instruction, to reduce
potential effects of their order, we randomly shuffle the candidates by $r$
times. Each of its permutation $Perm_{j}(Cand^{i}_{k})$,
$j\in\\{1,2,\dots,r\\}$, will be used to construct an individual tuple of
(Instruction, Query, Output) to fine-tune the LLM. The instruction has the
following format:

> Instruction: Given a set of candidate parent classes: $Perm_{j}(Cand^{k})$,
> output the most likely parent class for the entity given by user.
>
> Query: Find the parent for $s_{i}$.
>
> Output: The parent class is $s^{i}_{parent}$.

During inference, we can use the fine-tuned LLM to find the parent node of
each entity in the query set $\mathcal{S}^{+}=\\{s_{1},\dots,s_{M}\\}$. For
each $s_{i}$ in $\mathcal{S}^{Query}$, we select top-$k$ entities $T^{k}_{i}$
following eq. 2. Then, we formulate $s_{i}$ and $T^{k}_{i}$ into the above
Instruction and Query, which are fed into the fine-tuned LLM to get Output.

In brief, for Taxonomy Expansion, we use instruction-tuning to fine-tune the
model on the self-supervision data generated from the existing taxonomy, and
then make inference using the same instruction with the top-$k$ candidate
parents.

###  3.3 Instruction Tuning for Seed-Guided Taxonomy Construction

Given a seed taxonomy, Seed-Guided Taxonomy Construction aims to enrich a
given a taxonomy $\mathcal{T}$ by first finding more entities and then placing
them under the right parent. It naturally splits into two subtasks: (1)
expanding the entity sets to discovery new entities and (2) expanding the
taxonomy by finding the proper parent for each new entity. Formally, we define
the subtasks as follows:

  1. 1.

Entity Set Expansion (Finding siblings):
Given a taxonomy
$\mathcal{T}=(\mathcal{S}_{0},\mathcal{S}_{1},...,\mathcal{S}_{L})$ with the
root $\mathcal{S}_{0}=\\{\textrm{ROOT}\\}$ and $\mathcal{S}_{l}$ ($1\leq l\leq
L$) denoting the set of seeds at layer $l$, this task aims to expand the set
of seeds $\mathcal{S}_{l}=\\{s^{l}_{1},s^{l}_{2},...,s^{l}_{M}\\}$ through
discovering more entities
$\mathcal{S}_{l}^{+}=\\{s^{l}_{M+1},s^{l}_{M+2},...,s^{l}_{M+N}\\}$, where
they belonging to the class $\mathcal{S}_{0}$ and share the same granularity
as $\mathcal{S}_{l}$.

  2. 2.

Taxonomy Expansion (Finding parents):
Given a taxonomy
$\mathcal{T}=(\mathcal{S}_{0},\mathcal{S}_{1},...,\mathcal{S}_{L})$, and let
$\mathcal{S}_{l}$ denote the set of seeds at layer $l$ and
$\mathcal{S}_{l}^{+}=\\{s^{l}_{M+1},s^{l}_{M+2},...,s^{l}_{M+N}\\}$ denote the
newly expanded entities. We aim to find the right parent for each
$s^{l}_{j}\in\mathcal{S}_{l}^{+}$ among entities at layer $l-1$, which are
$\mathcal{S}_{l-1}=\\{s^{l-1}_{1},s^{l-1}_{2},...,s^{l-1}_{K}\\}$,
$K=|\mathcal{S}_{l-1}|$.

In Seed-Guided Taxonomy Construction, similar to Taxonomy Expansion, we are
also given a taxonomy structure $\mathcal{T}$ as input. Thus, we can also
generate self-supervision training data consisting of (Instruction Query
Output) tuples similar to the taxonomy expansion task to fine-tune the LLM
(c.f. section 3.2).

With the fine-tuned LLM, we first perform Entity Set Expansion, by following a
similar procedure as in Section 3.1: we generate permutations of seed
entities, formulate them into multiple (Instruction, Query) tuples, make
inference, and then use the auxiliary PLM to sort tuples’ output entities into
a ranked list $\mathcal{S}^{output}_{sorted}$. One subtle difference is that
we are usually given a meaningful ROOT node $\mathcal{S}_{0}$ that could be
directly used as the parent class, which means we do not need to prompt the
LLM to generate it. However, in some case where the ROOT node is a pseudo node
without an actual meaning, we can follow the procedure in Section 3.1 to
prompt the LLM to find a parent.

Then, for each entity $s_{i}\in\mathcal{S}^{output}_{sorted}$, we perform
Taxonomy Expansion to place each expanded entity in the right position in
$\mathcal{T}$. We use the same (Instruction, Query) as Taxonomy Expansion’s
and prompt the LLM to predict the parent $s^{i}_{parent}$. By attaching each
new entity to its parent node in $\mathcal{T}$, we get an expanded taxonomy
$\mathcal{T}^{+}$.

###  3.4 A Unified Framework

With the above three different schemas, one can directly fine-tune an LLM and
perform each task separately. However, a generally pre-trained LLM cannot
effectively capture the structural information, essentially the hypernymy and
sibling relations, among entities. The task specific data may also be too
small for the model to learn enough knowledge. For example, the seed taxonomy
in Seed-Guided Taxonomy Construction typically contains only around 10 nodes.

Therefore, we propose to first jointly instruction tune an LLM on an external
large taxonomy. We fine-tune the model on the Taxonomy Expansion task and
Entity Set Expansion task with the same self-supervision data described in
each task. In this way, the LLM can first learn to recognize and understand
the hypernymy and sibling relations before being applied into each task.

In this work, we use the CTD’s MEDIC disease vocabulary Davis et al. (2022) as
our external taxonomy. We can generate self-supervision data from the taxonomy
in the following way. Given a parent node $s_{parent}$ and its children
${s_{1},s_{2},\dots,s_{n}}$, we generate training data in two ways: (1) For
each child $s_{i}$, we ask the LLM to find its true parent $s_{parent}$ from a
set of candidates that are semantically similar to $s_{i}$ (c.f., equation 2).
(2) For subsets of the children
${s^{\prime}_{1},\dots,s^{\prime}_{4}}\subset{s_{1},s_{2},\dots,s_{n}}$, we
ask the LLM to expand this set to recover other siblings. We will then fine-
tune the LLM on the above self-supervision data using the aforementioned
Taxonomy Expansion’s and Entity Set Expansion’s instruction schemas.

After the unified fine-tuning on the external taxonomy, we will use the LLM on
each downstream task. It will be further fine-tuned with the task specific
data for the Taxonomy Expansion and Seed-Guided Taxonomy Construction tasks,
or directly used for inference on the Entity Set Expansion task.

##  4 Experiments

We conduct experiments on three tasks by comparing TaxoInstruct with strong
baselines in each task. Appendix A include the implementation details.

###  4.1 Entity Set Expansion

Dataset. Shen et al. present two versions of the PubMed-CVD dataset for
evaluating model performance in Entity Set Expansion Shen et al. (2017) and
Seed-Guided Taxonomy Construction Shen et al. (2018a), respectively, in the
scientific domain. The hierarchical version is still publicly accessible,
which will be used in Section 4.3. However, the flat version can no longer be
found. Hence, we rebuild a flat version of PubMed-CVD. Specifically, we
consider 3 classes of cardiovascular diseases, each of which has 5 seeds.
Because some baselines Shen et al. (2017); Zhang et al. (2020) rely on a large
corpus to learn text semantics, we extract more than 134K PubMed articles from
the MAPLE Medicine-MeSH dataset Zhang et al. (2023b) that are relevant to
cardiovascular diseases (according to their MeSH terms Coletti and Bleich
(2001)) as input to these baselines.

Baselines. We compare TaxoInstruct with the following Entity Set Expansion
methods: SetExpan Shen et al. (2017), CGExpan Zhang et al. (2020), SECoExpan
Zhang et al. (2022). Please see Appendix B.1 for more details of the
baselines. We also include an ablation of our method, TaxoInstruct-NoTune,
which directly prompts a pre-trained LLM to perform downstream tasks without
taxonomy-guided instruction tuning. We use the same LLM and instruction as our
full model. When experimenting with TaxoInstruct-NoTune, we discover that it
always generates fewer than 10 entities for each run. It also tends to produce
the same set of entities for different permutations of seeds. As a result, we
only report MAP@10 for TaxoInstruct-NoTune.

Evaluation Metrics. Following Shen et al. (2017) and Zhang et al. (2020), we
use the Mean Average Precision (MAP@$k$) as the evaluation metric, which
calculates the accuracy of top-$k$ expanded entities. We use $k=10$ and $k=20$
in our experiments. The full definition of MAP@$k$ is in Appendix B.2.

Table 1: Performance of compared methods in the Entity Set Expansion task. Bold: the best score. ‘–’: we cannot get 20 expanded entities because TaxoInstruct-NoTune generates highly similar answers with different permutations of seeds. Method | PubMed-CVD
---|---
MAP@10 | MAP@20
SetExpan | 59.68 | 43.93
CGExpan | 73.16 | 68.72
SECoExpan | 58.36 | 59.73
TaxoInstruct | 79.15 | 71.72
-NoTune | 69.80 | –

Experimental Results Table 1 shows the performance of compared methods in the
Entity Set Expansion task. We observe that TaxoInstruct achieves the best
performance in terms of both MAP@10 and MAP@20. This rationalizes our
motivation to formulate the goal of Entity Set Expansion as finding siblings.
Our model also outperform its ablated version TaxoInstruct-NoTune, which
demonstrates the effectiveness of unified pre-training on an external taxonomy
(c.f. Section 3.4).

###  4.2 Taxonomy Expansion

Datasets. Following Jiang et al. (2023), we use two public datasets,
Environment and Science, from the shared task in SemEval 2016 Bordea et al.
(2016). The considered entities (both existing ones in the input taxonomy and
new ones to be inserted) are scientific concepts related to environment and
general science, respectively. In both datasets, we adopt the same training-
testing split as in Yu et al. (2020); Jiang et al. (2023), where 20% of the
leaf nodes in the taxonomy are used for testing and the remaining nodes for
training.

Baselines We compare TaxoInstruct with the following Taxonomy Expansion
methods: TAXI Panchenko et al. (2016), HypeNET Shwartz et al. (2016), BERT+MLP
Devlin et al. (2019), TaxoExpan Shen et al. (2020), STEAM Yu et al. (2020),
BoxTaxo Jiang et al. (2023), and the ablation TaxoInstruct-NoTune as
introduced in Section 4.1. We only include baselines that have released their
code for reproducing the results. Please see Appendix C.1 for more details on
the baselines.

Evaluation Metrics. Following Yu et al. (2020); Jiang et al. (2023), we adopt
two metrics to evaluate the quality of taxonomy expansion results, namely
Accuracy (Acc) and Wu & Palmer Similarity (Wu&P) Wu and Palmer (1994). Acc
evaluates the predicted parent using exact match, while Wu&P uses the tree
distance between the predicted and ground truth parents. Please see Appendix
C.2 for their formal definitions.

Previous studies Yu et al. (2020); Jiang et al. (2023) also consider the mean
reciprocal rank (MRR) as an evaluation metric. However, this metric requires a
model to rank all nodes in the taxonomy according to their likelihood of being
the parent, thus it is not applicable to TaxoInstruct that is based on a
generative LLM.

Table 2: Performance of compared methods in the Taxonomy Expansion task. Scores of the baselines (except TaxoInstruct-NoTune) are reported in Yu et al. (2020); Jiang et al. (2023). Bold: The best score. *: TaxoInstruct is significantly better than this method with p-value $<0.01$. Method | Environment | Science
---|---|---
Acc | Wu&P | Acc | Wu&P
TAXI | 16.7∗ | 44.7∗ | 13.0∗ | 32.9∗
HypeNET | 16.7∗ | 55.8∗ | 15.4∗ | 50.7∗
BERT+MLP | 11.1∗ | 47.9∗ | 11.5∗ | 43.6∗
TaxoExpan | 11.1∗ | 54.8∗ | 27.8∗ | 57.6∗
STEAM | 36.1∗ | 69.6∗ | 36.5∗ | 68.2∗
BoxTaxo | 38.1∗ | 75.4 | 31.8∗ | 64.7∗
TaxoInstruct | 46.79 | 77.39 | 41.57 | 73.70
-NoTune | 21.15∗ | 65.30∗ | 12.94∗ | 62.87∗

Experimental Results. Table 2 shows the performance of compared methods in
taxonomy expansion. We run TaxoInstruct three times and report the average
performance. To show statistical significance, we conduct a two-tailed Z-test
to compare TaxoInstruct with each baseline. The significance level is marked
in Table 2. TaxoInstruct consistently and significantly outperforms all
baselines and also TaxoInstruct-NoTune. We first show the importance of
utilizing self-supervision, since directly prompt a LLM yield sub-optimal
result. Moreover, comparing with other baselines, we also show that we can use
self-supervision data most effectively through instruction tuning on the task
of finding the true parent among high ranked candidate parents.

###  4.3 Seed-Guided Taxonomy Construction

Dataset We adopt the PubMed-CVD dataset (the hierarchical version) introduced
in Shen et al. (2018a) and consider the same input seeds. To be specific, the
input taxonomy has 2 layers, with 3 seeds at the top layer (i.e.,
cardiovascular abnormalities, vascular diseases, and heart disease) and 3
seeds under each top-layer seed. In other words, there are $3+3\times 3=12$
seeds in total.

Baselines We compare TaxoInstruct with the following Seed-Guided Taxonomy
Construction baselines: HSetExpan Shen et al. (2017), HiExpan Shen et al.
(2018a), and TaxoInstruct-NoTune as introduced in Section 4.1. Shen et al.
(2018a) have released the output taxonomies222http://bit.ly/2Jbilte of
HSetExpan and HiExpan on PubMed-CVD, which are used by us for evaluating the
two models.

Evaluation Metrics At the top layer (i.e., coarse-grained categories of
cardiovascular diseases), both HiExpan and TaxoInstruct achieve 100% accuracy.
Therefore, our evaluation metrics focus on the more challenging bottom layer.
We use Sibling P@$k$ to evaluates the accuracy of the sibling-finding step and
Parent P@$k$ for the accuracy of the parent-finding step. The full definition
of Sibling P@$k$ and Parent P@$k$ is in Appendix D.2.

Table 3: Performance of compared methods in the Seed-Guided Taxonomy Construction task. Bold and *: the same meanings as in Table 2. ‘–’: similar to Table 1. Method | PubMed-CVD (Hierarchical)
---|---
| Sibling
---
P@20
| Sibling
---
P@40
| Parent
---
P@20
| Parent
---
P@40
HSetExpan | 80.0 | 65.0∗ | 60.0∗ | 50.0∗
HiExpan | 60.0∗ | 72.0∗ | 60.0∗ | 65.0
TaxoInstruct | 85.0 | 79.0 | 78.0 | 72.0
-NoTune | 50.0∗ | – | 45.0∗ | –

####  4.3.1 Experimental Results

Table 3 demonstrates the performance of compared methods in Seed-Guided
Taxonomy Construction. We run TaxoInstruct five times with the average scores
reported. We find that that TaxoInstruct achieves the best overall performance
in terms of finding the correct siblings (Sibling P@20, Sibling P@40) and
finding the correct the parent (Parent P@20, Parent P@40) This is consistent
with our conclusion draw from previous experiment on Taxonomy Expansion and
Entity Set Expansion (Section 4.1.4 and Section 4.2.4), demonstrating the
effectiveness of our framework.

###  4.4 Qualitative Results

In Figure 3, we show a part of the taxonomy generated by TaxoInstruct on the
Pubmed-CVD data for the Seed-Guided Taxonomy Construction task. The blue boxes
and yellow boxes represent seeds and expanded entities, respectively. We can
see that TaxoInstruct can successfully generate high quality entities, such as
"heart rupture" and "heart aneurysm", and place them under the right parent
"heart disease".

###  4.5 Hyperparameter Study

We study the effect of one hyperparameter on performance of TaxoInstruct: the
number of shuffles in the self-supervision data of the Taxonomy Expansion
task. The Wu&P scores of TaxoInstruct on Environment and Science are plotted
in Figure 4. We can find that the performance of TaxoInstruct improves when
the number of shuffles increase from 1 to 10, and becomes stable afterwards.
As shown in the figure, all TaxoInstruct’s Wu&P scores significantly
outperform a strong baseline STEAM when the number of shuffles is greater than
10.

Figure 3: Parts of the taxonomy constructed by TaxoInstruct on PubMed-CVD.
Blue boxes indicate seeds, and yellow boxes indicate expanded entities.

##  5 Related Work

Entity Set Expansion. The task of Entity Set Expansion is first introduced in
EgoSet Rong et al. (2016), and early work like Mamou et al. (2018) uses
distributional similarity to rank candidate entities according to the seed
entities. In order to prevent semantic drifting, the latter approaches like
SetExpan Shen et al. (2017) start to apply iterative bootstrapping to select
new context features and new entities. After the introduction of Pre-trained
Language Model, it has been used to replace distributional semantics
representation. CGExpan Zhang et al. (2020) uses it to automatically generate
class names as a stronger signal to prevent semantic drifting. Instead of just
using content embedding of entities, SECoExpan Zhang et al. (2022) also
leverages their context embeddings to generate more accurate entity
representation.

Taxonomy Expansion. Automatic taxonomy construction and expansion is an
important task in many NLP tasks and has been explored by many previous
literature. In early studies, distributional word representations Fu et al.
(2014) and lexical patterns Snow et al. (2004) have been used to find
hypernym. Later, many attempts have focused on summarizing either local ego-
graphs Shen et al. (2020) or mini-paths Yu et al. (2020) and encode them using
neural networks. Most recently, Jiang et al. (2023) use box embedding to
replace single vector embedding to better capture graph structure. Multiple
works have also used pre-trained language model to replace traditional neural
networks. Liu et al. (2021) propose to train BERT Devlin et al. (2019) on the
taxonomy on Dynamic Margin Loss and then employed to generate taxonomy path
and query representation. Xu et al. (2022) adopt prompt tuning on encoder
model to generate better embeddings for taxonomy’s global structure. Shen et
al. (2018a) introduce a more challenging version of Taxonomy Expansion: Seed-
Guided Taxonomy Construction, which requires the initial step of selecting a
subset of entities that align with the taxonomy’s content before performing
Taxonomy Expansion on the subset.

Figure 4: Parameter sensitivity analysis on the Taxonomy Expansion task.
Effect of the number of shuffles on Environment (left) and Science (right).

Structure-Guided Instruction Tuning. Inspired by the success of the large
language model (LLM) in various domains, there has been increasing attention
on using LLM to learn structured data and solve graph problems. Many
approaches attempt to combine LLM massive knowledge with Graph Neural
Network’s (GNN) awareness of structured data He et al. (2023); Qin et al.
(2023). Other approaches solely use LLM on graph problems, without the help of
a GNN module. Guo et al. (2023) use formal language to describe graphs to GPT.
Zhang et al. (2023a) incorporate textual embeddings and structure embeddings
to enable LLM to perform on knowledge graph completion task. Ye et al. (2023)
demonstrate that using LLM as the foundation model could outperform
competitive GNN baselines, and proposes to replace GNN with LLM entirely. Our
work is the first to explore LLM’s potential on the entity enrichment task,
and is also the first to use LLM and instruction tuning to establish a unified
framework.

##  6 Conclusions

In this paper, we propose TaxoInstruct, a unified framework for the three
representative entity enrichment task: Entity Set Expansion, Taxonomy
Expansion, and Seed-Guided Taxonomy Construction. We utilize taxonomy-guided
instruction tuning to effectively extract and exploit self-supervision data
from the existing taxonomy. We then build a unified framework through
leveraging the commonality of the three tasks. We also conduct extensive
experiments on the benchmark datasets, which have demonstrated the superiority
of TaxoInstruct over previous task-specific baselines.

## Limitations

We state the limitations of this paper from the following perspectives. 1)
Given the generative nature of large language model, we employ a corpus
independent approach for both Entity Set Expansion and Seed-Guided Taxonomy
Construction by directly generating new entities, while previous studies are
corpus-dependent by extracting new entities from domain corpora Shen et al.
(2017); Zhang et al. (2020); Shen et al. (2018a). Although TaxoInstruct can
outperform all baselines, it does not utilize the domain knowledge in the text
data, and may suffer in very specific domains.

2) Existing entity enrichment datasets assume that each node has only one
parent in the taxonomy. However, in the real applications, a taxonomy is
normally a directed acyclic graph instead of a simple tree, meaning that a
node van have multiple parents. How TaxoInstruct performs on a DAG taxonomy is
not evaluated.

## References

  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  Scibert: A pretrained language model for scientific text.  In _EMNLP’19_ , pages 3615–3620.
  * Bordea et al. (2016) Georgeta Bordea, Els Lefever, and Paul Buitelaar. 2016.  Semeval-2016 task 13: Taxonomy extraction evaluation (texeval-2).  In _SemEval’16_ , pages 1081–1091.
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. 2020.  Specter: Document-level representation learning using citation-informed transformers.  _arXiv preprint arXiv:2004.07180_.
  * Coletti and Bleich (2001) Margaret H Coletti and Howard L Bleich. 2001.  Medical subject headings used to search the biomedical literature.  _JAMIA_ , 8(4):317–323.
  * Davis et al. (2022) Allan Peter Davis, Thomas C Wiegers, Robin J Johnson, Daniela Sciaky, Jolene Wiegers, and Carolyn J Mattingly. 2022.  Comparative toxicogenomics database (ctd): update 2023.  _Nucleic acids research_ , 51(D1):D1257–D1262.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT’19_ , pages 4171–4186.
  * Fu et al. (2014) Ruiji Fu, Jiang Guo, Bing Qin, Wanxiang Che, Haifeng Wang, and Ting Liu. 2014.  Learning semantic hierarchies via word embeddings.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1199–1209.
  * Gonçalves et al. (2019) Rafael S Gonçalves, Matthew Horridge, Rui Li, Yu Liu, Mark A Musen, Csongor I Nyulas, Evelyn Obamos, Dhananjay Shrouty, and David Temple. 2019.  Use of owl and semantic web technologies at pinterest.  In _ISWC’19_ , pages 418–435.
  * Guo et al. (2023) Jiayan Guo, Lun Du, and Hengyu Liu. 2023.  Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking.  _arXiv preprint arXiv:2305.15066_.
  * He et al. (2023) Xiaoxin He, Xavier Bresson, Thomas Laurent, and Bryan Hooi. 2023.  Explanations as features: Llm-based features for text-attributed graphs.  _arXiv preprint arXiv:2305.19523_.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Jiang et al. (2022) Minhao Jiang, Xiangchen Song, Jieyu Zhang, and Jiawei Han. 2022.  Taxoenrich: Self-supervised taxonomy completion via structure-semantic representations.  In _Proceedings of the ACM Web Conference 2022_ , pages 925–934.
  * Jiang et al. (2023) Song Jiang, Qiyue Yao, Qifan Wang, and Yizhou Sun. 2023.  A single vector is not enough: Taxonomy expansion via box embeddings.  In _WWW’23_ , pages 2467–2476.
  * Liu et al. (2021) Zichen Liu, Hongyuan Xu, Yanlong Wen, Ning Jiang, Haiying Wu, and Xiaojie Yuan. 2021.  Temp: taxonomy expansion with dynamic margin loss through taxonomy-paths.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3854–3863.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Ma et al. (2023) Kaixin Ma, Hao Cheng, Yu Zhang, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. 2023.  Chain-of-skills: A configurable model for open-domain question answering.  In _ACL’23_ , pages 1599–1618.
  * Mamou et al. (2018) Jonathan Mamou, Oren Pereg, Moshe Wasserblat, Alon Eirew, Yael Green, Shira Guskin, Peter Izsak, and Daniel Korat. 2018.  Term set expansion based nlp architect by intel ai lab.  _arXiv preprint arXiv:1808.08953_.
  * Mao et al. (2020) Yuning Mao, Tong Zhao, Andrey Kan, Chenwei Zhang, Xin Luna Dong, Christos Faloutsos, and Jiawei Han. 2020.  Octet: Online catalog taxonomy enrichment with self-supervision.  In _KDD’20_ , pages 2247–2257.
  * Panchenko et al. (2016) Alexander Panchenko, Stefano Faralli, Eugen Ruppert, Steffen Remus, Hubert Naets, Cédrick Fairon, Simone Paolo Ponzetto, and Chris Biemann. 2016.  Taxi at semeval-2016 task 13: a taxonomy induction method based on lexico-syntactic patterns, substrings and focused crawling.  In _SemEval’16_ , pages 1320–1327.
  * Qin et al. (2023) Yijian Qin, Xin Wang, Ziwei Zhang, and Wenwu Zhu. 2023.  Disentangled representation learning with large language models for text-attributed graphs.  _arXiv preprint arXiv:2310.18152_.
  * Qu et al. (2018) Meng Qu, Xiang Ren, Yu Zhang, and Jiawei Han. 2018.  Weakly-supervised relation extraction by pattern-enhanced embedding learning.  In _WWW’18_ , pages 1257–1266.
  * Rong et al. (2016) Xin Rong, Zhe Chen, Qiaozhu Mei, and Eytan Adar. 2016.  Egoset: Exploiting word ego-networks and user-generated ontology for multifaceted set expansion.  In _WSDM’16_ , pages 645–654.
  * Shen et al. (2020) Jiaming Shen, Zhihong Shen, Chenyan Xiong, Chi Wang, Kuansan Wang, and Jiawei Han. 2020.  Taxoexpan: Self-supervised taxonomy expansion with position-enhanced graph neural network.  In _WWW’20_ , pages 486–497.
  * Shen et al. (2017) Jiaming Shen, Zeqiu Wu, Dongming Lei, Jingbo Shang, Xiang Ren, and Jiawei Han. 2017.  Setexpan: Corpus-based set expansion via context feature selection and rank ensemble.  In _ECML-PKDD’17_ , pages 288–304.
  * Shen et al. (2018a) Jiaming Shen, Zeqiu Wu, Dongming Lei, Chao Zhang, Xiang Ren, Michelle T Vanni, Brian M Sadler, and Jiawei Han. 2018a.  Hiexpan: Task-guided taxonomy construction by hierarchical tree expansion.  In _KDD’18_ , pages 2180–2189.
  * Shen et al. (2018b) Jiaming Shen, Jinfeng Xiao, Xinwei He, Jingbo Shang, Saurabh Sinha, and Jiawei Han. 2018b.  Entity set search of scientific literature: An unsupervised ranking approach.  In _SIGIR’18_ , pages 565–574.
  * Shen et al. (2018c) Zhihong Shen, Hao Ma, and Kuansan Wang. 2018c.  A web-scale system for scientific knowledge exploration.  In _ACL’18 System Demonstrations_ , pages 87–92.
  * Shwartz et al. (2016) Vered Shwartz, Yoav Goldberg, and Ido Dagan. 2016.  Improving hypernymy detection with an integrated path-based and distributional method.  In _ACL’16_ , pages 2389–2398.
  * Snow et al. (2004) Rion Snow, Daniel Jurafsky, and Andrew Ng. 2004.  Learning syntactic patterns for automatic hypernym discovery.  _Advances in neural information processing systems_ , 17.
  * Tabassum et al. (2020) Jeniya Tabassum, Mounica Maddela, Wei Xu, and Alan Ritter. 2020.  Code and named entity recognition in stackoverflow.  In _ACL’20_ , pages 4913–4926.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Vilnis et al. (2018) Luke Vilnis, Xiang Li, Shikhar Murty, and Andrew McCallum. 2018.  Probabilistic embedding of knowledge graphs with box lattice measures.  In _ACL’18_ , pages 263–272.
  * Wang et al. (2021) Qingyun Wang, Manling Li, Xuan Wang, Nikolaus Parulian, Guangxing Han, Jiawei Ma, Jingxuan Tu, Ying Lin, Haoran Zhang, Weili Liu, et al. 2021.  Covid-19 literature knowledge graph construction and drug repurposing report generation.  In _NAACL’21 System Demonstrations_ , pages 66–77.
  * Wang and Cohen (2007) Richard C Wang and William W Cohen. 2007.  Language-independent set expansion of named entities using the web.  In _ICDM’07_ , pages 342–350.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022.  Finetuned language models are zero-shot learners.  In _ICLR’22_.
  * Wu and Palmer (1994) Zhibiao Wu and Martha Palmer. 1994.  Verbs semantics and lexical selection.  In _ACL’94_ , pages 133–138.
  * Xu et al. (2022) Hongyuan Xu, Yunong Chen, Zichen Liu, Yanlong Wen, and Xiaojie Yuan. 2022.  Taxoprompt: A prompt-based generation method with taxonomic context for self-supervised taxonomy expansion.
  * Ye et al. (2023) Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, and Yongfeng Zhang. 2023.  Natural language is all a graph needs.  _arXiv preprint arXiv:2308.07134_.
  * Yu et al. (2020) Yue Yu, Yinghao Li, Jiaming Shen, Hao Feng, Jimeng Sun, and Chao Zhang. 2020.  Steam: Self-supervised taxonomy expansion with mini-paths.  In _KDD’20_ , pages 1026–1035.
  * Zhang et al. (2023a) Yichi Zhang, Zhuo Chen, Wen Zhang, and Huajun Chen. 2023a.  Making large language models perform better in knowledge graph completion.  _arXiv preprint arXiv:2310.06671_.
  * Zhang et al. (2023b) Yu Zhang, Bowen Jin, Qi Zhu, Yu Meng, and Jiawei Han. 2023b.  The effect of metadata on scientific literature tagging: A cross-field cross-model study.  In _WWW’23_ , pages 1626–1637.
  * Zhang et al. (2022) Yu Zhang, Yunyi Zhang, Yucheng Jiang, Martin Michalski, Yu Deng, Lucian Popa, ChengXiang Zhai, and Jiawei Han. 2022.  Entity set co-expansion in stackoverflow.  In _IEEE BigData’22_ , pages 4792–4795.
  * Zhang et al. (2020) Yunyi Zhang, Jiaming Shen, Jingbo Shang, and Jiawei Han. 2020.  Empower entity set expansion via language model probing.  In _ACL’20_ , pages 8151–8160.

##  Appendix A Implementation Details

We use LLaMA-7B Touvron et al. (2023) as our base model, fine-tuned it with
LoRA Hu et al. (2021), which utilizes low rank matrix for parameter-efficient
fine-tuning. During fine-tuning, We fine-tuned about 0.6 % of trainable
parameters, which is about 40 million parameters. We use the cross entropy
loss on the tokens output in Output, but not Instruction and Query. The
training batch size is 64 for both pre-training and task-specific fine-tuning.
The optimizer is AdamW Loshchilov and Hutter (2017). During pre-training, we
train on CTD’s data Davis et al. (2022) for 20 epochs, which took about 3
hours using two NIVIDIA RTX A6000 GPUs. During fine-tuning, in Seed-Guided
Taxonomy Construction, since we are given a small set of seeds, we generate
all permutations of the seeds when creating self-supervision data and fine-
tune for 10 epochs, which took about 30 minutes. In Taxonomy Expansion, we
shuffle the candidate parents for 10 times and train on self-supervision data
for 5 epochs. During inference, in Taxonomy Expansion, we adapt SPECTER Cohan
et al. (2020) as the auxiliary PLM to select top-20 candidate parents. In
Entity Set Expansion and Seed-Guided Taxonomy Construction, in order to
generate sufficient new entities, we keep shuffling the input seed entities
and make inference using the permuted set of entities until we have generated
sufficient new entities (number of new entities $>$ 400) or we have shuffled
for sufficient amount of times (number of shuffles $>$ 50).

##  Appendix B Experiment Setup for Entity Set Expansion

###  B.1 Baselines

We compare TaxoInstruct with the following entity set expansion methods.

  * •

SetExpan Shen et al. (2017)333https://github.com/mickeysjm/SetExpan
iteratively selects skip-gram context features from the corpus and proposes a
rank ensemble mechanism for scoring and selecting entities.

  * •

CGExpan Zhang et al. (2020)444https://github.com/yzhan238/CGExpan infers the
target semantic class names by probing a pre-trained language model and then
utilizes the generated class names to expand new entities. In the original
paper, BERT Devlin et al. (2019) is used as the language model. Here, we
replace BERT with SciBERT Beltagy et al. (2019) and observe better performance
on PubMed-CVD.

  * •

SECoExpan Zhang et al. (2022)555https://github.com/yuzhimanhua/SEType expands
multiple entity sets simultaneously according to the contents and contexts of
each entity. It imposes mutual exclusivity on the expanded entities. Here, we
replace the originally used BERTOverflow Tabassum et al. (2020) with SciBERT
Beltagy et al. (2019) and observe better performance on PubMed-CVD.

###  B.2 Evaluation Metrics

We use the Mean Average Precision (MAP@$k$) to evaluate the entity set
expansion results. Formally, given a set of seeds
$\mathcal{S}=\\{s_{1},...,s_{M}\\}$ and the top-$k$ expanded entities
$\mathcal{S}^{+}=\\{s_{M+1},...,s_{M+k}\\}$, the average precision
$\textrm{AP}@k$ is defined as

$\textrm{AP}@k(\mathcal{S},\mathcal{S}^{+})=\frac{1}{k}\sum_{u:1\leq u\leq k\
\land\atop
s_{M+u}\sim\mathcal{S}}\frac{\sum_{v=1}^{u}\mathbb{I}(s_{M+v}\sim\mathcal{S})}{u}.$
(3)

Here, $s_{M+u}\sim\mathcal{S}$ means that the expanded entity $s_{M+u}$ and
the seeds $\mathcal{S}$ belong to the same semantic class. Since there are
multiple testing queries (i.e., multiple sets of seeds)
$\mathcal{S}_{1},...,\mathcal{S}_{C}$ and their corresponding expansion
results $\mathcal{S}_{1}^{+},...,\mathcal{S}_{C}^{+}$, MAP@$k$ is defined as

$\textrm{MAP}@k=\frac{1}{C}\sum_{i=1}^{C}\textrm{AP}@k(\mathcal{S}_{i},\mathcal{S}_{i}^{+}).$
(4)

##  Appendix C Experiment Setup for Taxonomy Expansion

###  C.1 Baselines

We compare TaxoInstruct with the following taxonomy expansion methods.

  * •

TAXI Panchenko et al. (2016)666https://github.com/uhh-lt/taxi is a taxonomy
induction method based on hypernymy detection. It first extracts hypernym-
hyponym pairs from domain-specific corpora using substrings and lexico-
syntactic patterns and then organizes the extracted terms into a coherent
taxonomy.

  * •

HypeNET Shwartz et al. (2016)777https://github.com/vered1986/HypeNET is a
taxonomy induction method which employs LSTM to concurrently capture the
distributional and relational information between term pairs along dependency
paths.

  * •

BERT+MLP Devlin et al. (2019) is a taxonomy induction method built upon pre-
trained BERT embeddings. It first acquires term embeddings from a pre-trained
BERT model and then input the embeddings into a multi-layer perceptron to
predict the hypernymy relationship.

  * •

TaxoExpan Shen et al. (2020)888https://github.com/mickeysjm/TaxoExpan is a
taxonomy expansion method which leverages graph neural networks to encode
local ego-graphs in the input taxonomy to improve entity representations. In
the original paper, context-free word embeddings are used as input features.
Following Yu et al. (2020), we replace context-free embeddings with more
powerful BERT embeddings for this baseline.

  * •

STEAM Yu et al. (2020)999https://github.com/yueyu1030/STEAM is a taxonomy
expansion method which learns representations for each pair of (new entity,
existing entity) from multiple views using paths sampled from the taxonomy.

  * •

BoxTaxo Jiang et al. (2023)101010https://github.com/songjiang0909/BoxTaxo is a
taxonomy expansion method which represents entities as boxes to capture their
parent-child relationship. It optimizes the box embedding Vilnis et al. (2018)
of each entity from a joint view of geometry and probability.

###  C.2 Evaluation Metrics

We adopt the following two evaluation metrics for the taxonomy expansion task.

  * •

Accuracy (Acc) is the exact match accuracy of the predicted parent node of
each testing entity. Formally, assume the testing set has $C$ samples
$x_{1},...,x_{C}$, and their ground-truth parents in the input taxonomy are
$y_{1},...,y_{C}$, respectively. Then the accuracy of the learned parent-child
relationship $\textsc{Parent}^{+}(\cdot)$ is defined as

$\textrm{Acc}=\frac{1}{C}\sum_{i=1}^{C}\mathbb{I}(\textsc{Parent}^{+}(x_{i})=y_{i}).$
(5)

  * •

Wu & Palmer Similarity (Wu&P) Wu and Palmer (1994) calculates the similarity
between the predicted parent and the ground-truth parent based on their
distance in the taxonomy.

$\textrm{Wu\&P}=\frac{1}{C}\sum_{i=1}^{C}\frac{2\times\textrm{depth}(\textrm{LCP}(\textsc{Parent}^{+}(x_{i}),y_{i}))}{\textrm{depth}(\textsc{Parent}^{+}(x_{i}))+\textrm{depth}(y_{i})},$
(6)

where $\textrm{LCP}(\cdot,\cdot)$ is the lowest common ancestor of two nodes,
and $\textrm{depth}(\cdot)$ denotes the depth of a node in the taxonomy.

##  Appendix D Experiment Setup for Seed-Guided Taxonomy Construction

###  D.1 Baselines

We compare TaxoInstruct with the following Seed-Guided Taxonomy Construction
methods.

  * •

HSetExpan Shen et al. (2017) iteratively applies SetExpan at each layer of the
input taxonomy. For each expanded bottom-layer node, it uses REPEL Qu et al.
(2018), a weakly supervised relation extraction model, to find the most proper
parent at the top layer.

  * •

HiExpan Shen et al. (2018a)111111https://github.com/mickeysjm/HiExpan combines
the techniques of flat set expansion, parent-child relationship inference, and
global optimization of the taxonomy structure. It jointly utilizes skip-grams,
context-free text embeddings, and entity types.

###  D.2 Evaluation Metrics

For the Seed-Guided Taxonomy Construction task, we adopt Sibling P@$k$ and
Parent P@$k$ to measure the quality of the sibling-finding step and parent-
finding step respectively.

  * •

Sibling P@$k$ evaluates the accuracy of the sibling-finding step. Given the
bottom-layer seeds $\mathcal{S}_{2}=\\{s_{2,1},...,s_{2,M}\\}$, we check the
top-$k$ expanded bottom-layer entity
$\mathcal{S}_{2}^{+}=\\{s_{2,M+1},...,s_{2,M+k}\\}$ and judge whether
$s_{2,M+i}$ and $\mathcal{S}_{2}$ belong to the same semantic class (i.e.,
whether $s_{2,M+i}$ is a cardiovascular disease).

$\textrm{Sibling
P@}k=\frac{1}{k}\sum_{i=1}^{k}\mathbb{I}(s_{2,M+i}\sim\mathcal{S}_{2}).$ (7)

  * •

Parent P@$k$ evaluates the accuracy of the parent-finding step. For each
expanded bottom-layer entity $s_{2,M+i}$, let $s_{1,p(i)}$ denote its ground-
truth parent at the top layer. Then, this metric can be defined as

$\textrm{Parent
P@}k=\frac{1}{k}\sum_{i=1}^{k}\mathbb{I}(\textsc{Parent}^{+}(s_{2,M+i})=s_{1,p(i)}).$
(8)