#  A Solution-based LLM API-using Methodology
for Academic Information Seeking

Yuanchun Wang†111, Jifan Yu§111Work was done when these authors interned at
Zhipu AI., Zijun Yao§, Jing Zhang†, Yuyang Xie§, Shangqing Tu§
Yiyang Fu†, Youhe Feng†, Jinkai Zhang†, Jingyao Zhang⋄, Bowen Huang⋄, Yuanyao
Li⋄
Huihui Yuan⋄, Lei Hou§, Juanzi Li§ and Jie Tang§
†Renmin University of China §Tsinghua University ⋄Zhipu AI
https://soay.aminer.cn

###### Abstract

Applying large language models (LLMs) for academic API usage shows promise in
reducing researchers’ academic information seeking efforts. However, current
LLM API-using methods struggle with complex API coupling commonly encountered
in academic queries. To address this, we introduce SoAy, a solution-based LLM
API-using methodology for academic information seeking. It uses code with a
solution as the reasoning method, where a solution is a pre-constructed API
calling sequence. The addition of the solution reduces the difficulty for the
model to understand the complex relationships between APIs. Code improves the
efficiency of reasoning.

To evaluate SoAy, we introduce SoAyBench, an evaluation benchmark accompanied
by SoAyEval, built upon a cloned environment of APIs from AMiner. Experimental
results demonstrate a 34.58-75.99% performance improvement compared to state-
of-the-art LLM API-based baselines. All datasets, codes, tuned models, and
deployed online services are publicly accessible at
https://github.com/RUCKBReasoning/SoAy.

A Solution-based LLM API-using Methodology
for Academic Information Seeking

Yuanchun Wang††††, Jifan Yu§111Work was done when these authors interned at
Zhipu AI., Zijun Yao§, Jing Zhang†, Yuyang Xie§, Shangqing Tu§ Yiyang Fu†,
Youhe Feng†, Jinkai Zhang†, Jingyao Zhang⋄, Bowen Huang⋄, Yuanyao Li⋄ Huihui
Yuan⋄, Lei Hou§, Juanzi Li§ and Jie Tang§ †Renmin University of China
§Tsinghua University ⋄Zhipu AI https://soay.aminer.cn

##  1 Introduction

Researchers frequently seek academic information such as a paper’s metadata, a
scholar’s publication record, and the connections between scholars as part of
their routine scholarly activities. The advent of academic information
retrieval systems—including DBLP dblp team (2005), CiteSeer Giles et al.
(1998), and Google Scholar Noruzi (2005)—and more sophisticated academic
mining platforms such as Microsoft Academic Search Sinha et al. (2015) and
AMiner Tang et al. (2008), has substantially reduced the complexity of
exploring the academic data, which encompasses massive entities like authors,
conferences, and papers.

Figure 1: Comparing responses from Chameleon Lu et al. (2023), ToolLLaMA Qin
et al. (2023b), and the proposed SoAy, all leveraging AMiner APIs to address
the same academic query.

Large Language Models Zhao et al. (2023) (LLMs) unveil the potential to
further reduce the effort required for academic information seeking,
benefiting from LLMs’ reasoning ability to comprehend intentions expressed in
natural language. Enabling LLMs to utilize existing academic retrieval/mining
APIs Qin et al. (2023a) tends to be a feasible solution. However, this
adaptation is not easy because of two main challenges:

(1) Coupling:  Academic information for academic queries is distributed across
various storage locations but interconnected internally, resulting in multiple
academic APIs responsible for different functions yet interconnected, i.e.
coupled. This requires LLM to fully understand the sequential order of calls
between these APIs. Existing API-using methods like Chameleon Lu et al.
(2023), shown in Figure 1(a), retrieve and execute APIs separately to get
results, thus failing to handle such API coupling.

(2) Efficiency:  Existing methods such as ToolLLM Qin et al. (2023b)
illustrated in Figure 1(b) employ a depth-first search-based decision tree
(DFSDT) for step-by-step reasoning, which has the potential to capture the
regularities in the sequence of API. However, the excessive steps lead to
query times that users may find unacceptable, compared with traditional
academic search engines that offer researchers a very fast response.

We propose SoAy, a Solution-based API-using methodology to apply LLMs to for
academic information seeking. Our key innovation lies in the new API calling
method: solution and code. As shown in Figure 1(c), SoAy first lets the LLM
generate a feasible API calling plan, i.e. solution, based on complex user
inputs, and then allows the LLM to generate executable API calling code based
on the generated solution. The pre-generated solutions can assist the LLM in
clarifying the complex coupling relationships between academic APIs when
generating subsequent code. Calling APIs through code generation in a single
inference process without repeated reasoning can achieve efficiency levels
closer to traditional academic search systems. Specifically, the ability for
LLM to reliably utilize complex academic APIs in this manner to accomplish
academic search tasks benefits from the automatic generation of a training
dataset, consisting of (query, solution, code) triplets. The data is created
on the AMiner APIs, resulting in 3,960 automated triplets. We retain 4/5 of
these triplets, providing valuable resources for fine-tuning existing LLMs or
demonstrating LLM in-context learning. This alignment helps bridge the gap
between general LLMs and the intricacies of academic API usage.

To evaluate SoAy, we utilize 1/5 of the generated data, comprising 792 (query,
solution, code) triplets. These are enhanced through human verification to
ensure the quality of the test set. We then introduce SoAyEval, an evaluation
methodology that takes into account both solution correctness and code
execution accuracy. By employing SoAyEval for automated evaluation alongside
human assessment, our experiments demonstrate the effectiveness and efficiency
of SoAy in utilizing AMiner APIs compared to state-of-the-art LLM API-using
baselines, across both in-context learning and tuning modes. Furthermore, we
validate the versatility of SoAy for application in other academic retrieval
systems.

The paper contributes: (1) SoAy, a novel LLM adaptation method, addressing
challenges like coupling and efficiency in academic information seeking
through automatic data generation, teaching LLMs to use academic APIs. (2)
SoAyBench, a released benchmark dataset, accompanied by the SoAyEval
evaluation method, to assess LLMs’ proficiency in using academic APIs. (3)
Demonstrated effectiveness and efficiency of SoAy in the AMiner system, along
with its real deployment on an online service as shown in Figure 13.

Figure 2: Illustration of AMiner API Library, API Graph and solution library.

##  2 Task Formulation

The task for SoAy is to seek appropriate academic information $I$ based on
complicated user query $Q$ in an efficient way. This objective tends to be
achieved by using academic APIs in an valid and efficient order.

###### Problem 1

User Intent Parse Based on $Q$, SoAy needs to understand which academic
information the user requires, then identify the associations within them and
conduct separate queries.

$\displaystyle Q\rightarrow Q_{S},\quad Q_{S}=\\{Q_{i}\\}_{i=1}^{N}.$ (1)

Traditional academic search systems cannot accomplish this task, thus
requiring users to break their requirements $Q$ down into step-by-step queries
$Q_{S}$ to type in the search box. SoAy, on the other hand, needs to use LLMs
to break the intent down and get the relevant API calling sequence.

###### Problem 2

Academic Information Aggregation After understanding what information the user
truly needs, SoAy’s task is to retrieve relevant information from data sources
according to $Q_{S}$ and aggregate and process the results to provide the user
with the precise information $I$ they seek.

Traditional information retrieval systems lack this design and instead display
all retrieved information to users for them to select what they want in a
list-like format. What SoAy needs to do is generate the correct code to call
the academic APIs and process the obtained information within a comparable
time to those traditional systems.

Figure 3: The overall process of SoAy around one case of an academic question.
The main steps of data generation are shown on the left. Two mode of LLM
alignment with the automated triplet dataset are on the right.

##  3 SoAy: Solution-based API-using

We introduce SoAy to adapt general LLMs for academic information seeking,
emphasizing the use of existing academic APIs. To achieve this, abundant
training data in the form of (academic query, executable API sequence) pairs
is essential, facilitating either fine-tuning of existing LLMs or guiding them
through in-context learning. Illustrated in the left part of Figure 3, this
data generation pipeline comprises four key steps:

(1) Solution Library Construction.  Utilizing the provided academic APIs, we
construct a graph representing the dependencies among APIs, and traverse this
graph to identify valid and simple paths, referred to as solutions. These
solutions collectively form a library known as the solution library.

(2) Combination Formulation.  To obtain a more precise semantic of each
solution in the library, we combine all the possible inputs of the initial API
and all the outputs of the final API of the solution to formulate a series of
combinations.

(3) Query Generation.  We employ ChatGPT to transform each combination into a
template question lacking specific academic entity data. These templates are
then diversified into various expressions and populated into potential
academic queries with data from a specialized academic source, the AMiner
database here.

(4) Code Generation. Finally, we use ChatGPT again to write a piece of
corresponding code given the combination, which is expected to correspond to
the template query that shares the same combination. Lastly, we employ ChatGPT
once more to generate the corresponding code for the given combination. This
code is anticipated to match the template query that has the same combination.

Notably, in the data preparation part, we indicate ChatGPT as
gpt-3.5-turbo-0613. We present the details of the data preparation prompt and
the process of setting up SoAyBench in the appendix.

##  4 Solution Library Construction

Given an academic API Library $L_{A}$, the initial step is to build an API
Graph where each node represents an API $f$, and a directed edge from node
$f_{n}$ to node $f_{m}$ signifies a coupling relationship from $f_{n}$ to
$f_{m}$ with the intersected attributes $a_{n,m}$.

###### Definition 1

API Coupling defines the frequent dependencies of the inputs and outputs in
academic API calling, caused by academic entity correlations’ complexity:

$\displaystyle f_{n}$ $\displaystyle\xrightarrow{a_{n,m}}f_{m},$ (2)
$\displaystyle if\quad a_{n,m}=i_{m}$ $\displaystyle\cap
o_{n}\quad\text{and}\quad a\neq\emptyset,$

where $i_{n}$ denotes the input parameters of $f_{m}$ and $o_{n}$ denotes the
returned attributes of $f_{n}$.

The API coupling $f_{n}\xrightarrow{a}f_{m}$ means that we can use $f_{n}$ to
output the attribute n,m, and subsequently use $a_{n,m}$ as the input
parameter to call $f_{m}$. This is because $a_{n,m}$ is the intersection of
the returned attributes $o_{n}$ of $f_{n}$ and the input parameters $i_{m}$ of
$f_{m}$. An example of API coupling, “searchPerson
$\xrightarrow{\text{person\\_id}}$ getPersonPubs” is illustrated in Figure
2(a) within academic APIs. The $G$ built from AMiner APIs is showed in Figure
2(b). By traversing this graph while limiting the maximum number of hops to
$H$, we can derive an API execution sequence, referred to as a solution:

###### Definition 2

A Solution $S$ is defined as an execution sequence of APIs traversed from the
API graph:

$\displaystyle S=f_{1}\xrightarrow{a_{1,2}}f_{2}\cdots\xrightarrow{a_{k-1,k}}$
$\displaystyle f_{k}\cdots\xrightarrow{a_{J-1,J}}f_{J},$ $\displaystyle\forall
k,f_{k}\in$ $\displaystyle L_{A}.$ (3)

A solution library $L_{S}$ denotes a set of valid and simple solutions,
ensuring by the following ways.

Valid Solution.  Among all traversed solutions from the API graph, validity is
determined by starting with a fuzzy API, as only fuzzy APIs can accept user
utterances in natural language as input.

Simple Solution.  For a single query, there may be multiple solutions
available. The preferred choice is the simplest one, containing the fewest
APIs. For instance, when inquiring about the publication date of the
representative work of a scholar, the optimal solution is “searchPerson
$\rightarrow$ getPersonPubs”, despite obtaining the same information is also
possible through “searchPerson $\rightarrow$ getPersonPubs $\rightarrow$
getPublication $\rightarrow$ searchPublication”.

We meticulously craft prompts to instruct ChatGPT in fulfilling the above two
characteristics. All prompts for question generation are detailed in Appendix
Part III. Prompts. Consequently, we’ve created 16 solutions to establish the
solution library $L_{S}$ for AMiner APIs. Figure 2(c) presents a subset of
these solutions, including their input parameters, returned attributes, and
corresponding question templates.

##  5 Reasoning Code Generation

In order for LLM to aggregate effective academic information and provide users
with precise answers, SoAy let LLM generate code containing API calls for
academic information retrieval. To achieve this, a large amount of Query,
Corresponding Code data needs to be generated to train LLM.

###  5.1 Combination Formulation

Section 2 describes the one-to-several relationship between a query and
multiple solutions. It’s evident that a similar one-to-several relationship
exists between a solution and several query intents. In a solution $S$, the
first API is referred to as the head API $f_{1}$, while the last API is termed
the tail API $f_{J}$. For example, in Figure 2(c), consider the 3-hop solution
“searchPublication $\rightarrow$ getPublication $\rightarrow$
getPersonBasicInfo”. Due to the diversity of the Head API Input $i_{1}$ and
the Tail API Output $o_{J}$, determining which information (outputs) is sought
under what conditions (inputs) remains uncertain even after the solution is
established. An intuitive approach is to combine the head API input, the
solution, and the tail API output to refine the intent space into a specific
requirement. A solution-based combination is defined as a specific binding of
the head API input, the solution, and the tail API output:
$C=(i_{1},S,o_{J})$. Given the $i_{1}$ is “title” and the $o_{J}$ is
“education_experience”, the $S$ “searchPublication $\rightarrow$
getPublication $\rightarrow$ getPersonBasicInfo” could lead to one plausible
interpretation: querying the graduation school of the first author of the
provided publication. It’s important to recognize that the information from
the academic data source corresponding to $i_{1}$ is both the information
needed by q-t to fill and the input information needed by the generated code,
$o_{J}$ is both the expected answer of q-t and the golden result of the code.

###  5.2 Query Generation

Template-Query Generation.  For a given combination $C$, we define a template
question $Q_{T}$ to represent the intent of $C$ with missing entity
information. In the example provided, the corresponding template question
$Q_{T}$ could be formulated as “Which school did the first author of XXXX
graduate from?”, where “XXXX” refers to a placeholder representing the title
of a publication, as illustrated in Table 1.

Semantic Expansion and Entity Filling.  A specific intent represented by a
template question $Q_{T}$ may be expressed by multiple natural language
queries. For instance, “Which school did XXX graduate from?” and “What is
XXX’s alma mater?” convey the same intent. To enrich the semantic diversity of
an intention, we conduct semantic augmentation by crafting prompts to leverage
ChatGPT in generating three new, semantically enriched template questions for
each combination.

Following semantic augmentation, we replace the special symbols in the
templates (represented by uppercase “X”s) with information about a specific
academic entity in an academic data soruce. A placeholder in a question
template must be replaced by the attributes of specific academic entities,
such as person_id, name, interest, organization, bio, educational_experience,
etc., as illustrated in Figure 2. Taking the instance of the alma mater query
again, The name of this scholar will be used to fill in the $Q_{T}$ to become
a real question “What is scholar name’s alma mater” and the educational
experience of this scholar serves as the golden answer to this query.

Table 1: The semantics of symbols in $Q_{T}$. Symbol | Semantics
---|---
X | interest
XX | organization
XXX | name
XXXX | publication_info
N | count or numbers
Y | Year

###  5.3 Code Generation

In the final step, we employ ChatGPT again to generate a piece of code that
corresponds to a given combination $C$ with missing input information. This
code is intended to get the answer of the template query $Q_{T}$ shared the
same combination $C$ with it. Still with information from a specific academic
entity, the $i_{1}$ serves as the input of the generated code. The code is
expected to be executed to get the same result as $o_{J}$, which can be seen
as a golden information of this code. If the result matches the golden
information, this code is valid. Any invalid code, not producing matching
results, will be excluded from the final dataset.

##  6 LLMs Alignment

In order to ultimately leverage LLM to accomplish the task of academic
information seeking, SoAy needs to align LLM to be capable of calling academic
APIs correctly and efficiently through generating solutions and code. This can
be achieved by constructing training data and aligning the LLMs according to
it, whether through a tuning or tuning-free method.

###  6.1 SoAyBench–{(Query, Solution, Code)}

Table 2: Question statistics in SoAyBench set.

Question Type | One-hop | Two-hop | Three-hop | Total
---|---|---|---|---
Scholar | 540 | 1,800 | 540 | 2,980
Publication | 180 | 180 | 720 | 1,080
Total | 720 | 1,980 | 1,230 | 3,960

After going through the aforementioned four stages, a substantial amount of
high-quality (Query, Solution, Code) data has been automatically collected. We
organize this data into a comprehensive benchmark known as SoAyBench.
Currently, SoAyBench consists of 3,960 triplets, out of which one-fifth has
undergone expert inspection and adjustment to form a test set. This test set
is used to evaluate the ability of the LLMs to search academic information
using the AMiner API. The remaining portion of the data is utilized to support
the alignment of the LLMs. Table 2 shows the question statistics of SoAyBench.

Table 3:  Overall Performance comparison of the proposed SoAy and the baseline
methods ToolLLaMA and GTP-DFSDT, as tested on SoAyBench. The terms 3.5,
3.5-16K, and 4 are abbreviations for different versions of GPT. 7B and 13B
denotes the parameter scale of the LLM. SoAyBench is derived from 44 question
templates, we calculate each metric within a group of instantiated questions
of these templates, and then report the mean and variance across groups with
different instantiated entities.

Method | Version | Question Type | Error Rate$\downarrow$ | EM(%) | ACC(%) | Score
---|---|---|---|---|---|---
DS(%) | WS(%) | WP(%) | EE(%)
ToolLLaMA | 7B | one-hop | 12.50±8.00 | 24.31±13.26 | 1.39±0.00 | 54.17±16.01 | 7.64±5.20 | 20.14 | 16.72
two-hop | 10.10±4.10 | 47.22±12.28 | 0.76±2.27 | 38.13±9.62 | 3.79±2.92 | 13.89
three-hop | 11.51±6.53 | 38.10±14.27 | 1.19±3.57 | 43.25±13.07 | 5.95±4.59 | 17.46
GPT-DFSDT | 3.5 | one-hop | 55.56±21.06 | 15.28±7.80 | 4.86±0.00 | 21.53±10.67 | 2.78±0.00 | 58.33 | 43.22
two-hop | 29.55±11.47 | 34.34±9.23 | 4.29±3.64 | 25.76±8.65 | 6.06±4.11 | 35.61
three-hop | 38.10±15.09 | 28.57±11.35 | 3.17±2.50 | 25.00±8.87 | 5.16±6.19 | 43.25
3.5-16k | one-hop | 25.69±10.91 | 9.72±5.00 | 2.78±0.00 | 22.92±9.47 | 38.89±15.60 | 64.58 | 43.67
two-hop | 16.92±7.76 | 15.91±6.05 | 3.28±1.31 | 46.97±7.13 | 16.92±4.99 | 33.84
three-hop | 18.65±7.37 | 15.48±5.63 | 2.78±0.00 | 38.49±10.43 | 24.60±8.53 | 43.25
4 | one-hop | 27.78±9.60 | 2.08±0.00 | 4.17±5.00 | 28.47±6.82 | 37.50±10.91 | 65.28 | 58.16
two-hop | 26.26±8.89 | 9.60±4.88 | 17.93±5.40 | 15.15±5.39 | 31.06±9.12 | 57.32
three-hop | 22.22±8.65 | 7.54±4.46 | 17.06±6.96 | 19.05±6.45 | 34.13±9.87 | 56.35
SoAyGPT | 3.5 | one-hop | 27.78±8.70 | 15.97±7.73 | 3.47±0.00 | 13.19±7.80 | 39.58±9.12 | 67.36 | 67.30
two-hop | 33.84±4.94 | 9.60±4.75 | 6.06±2.81 | 13.13±7.12 | 37.37±5.06 | 71.21
three-hop | 22.22±6.43 | 12.70±5.91 | 9.52±4.42 | 13.10±6.72 | 42.46±6.00 | 64.68
3.5-16k | one-hop | 28.47±11.67 | 15.28±6.12 | 1.39±0.00 | 17.36±7.78 | 37.50±9.07 | 65.97 | 66.76
two-hop | 35.86±6.01 | 7.32±3.41 | 5.30±2.18 | 15.91±7.16 | 35.61±4.65 | 71.46
three-hop | 23.02±7.16 | 10.32±4.99 | 8.33±3.42 | 17.46±7.37 | 40.87±6.26 | 63.89
4 | one-hop | 0.00±0.00 | 0.00±0.00 | 1.39±0.00 | 2.78±0.00 | 95.83±5.70 | 95.83 | 86.57
two-hop | 15.91±4.71 | 1.26±0.00 | 9.34±1.07 | 2.02±1.69 | 71.46±3.74 | 87.37
three-hop | 6.75±0.00 | 0.40±0.00 | 14.68±1.68 | 1.98±0.00 | 76.19±3.25 | 82.94
SoAyLLaMA | Chat-7B | one-hop | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | 0.69±0.00 | 99.31±2.94 | 99.31 | 85.76
two-hop | 0.00±0.00 | 0.00±0.00 | 20.20±3.84 | 2.53±1.97 | 77.27±2.70 | 77.27
three-hop | 0.00±0.00 | 0.00±0.00 | 9.92±3.56 | 3.17±2.50 | 86.90±2.72 | 86.90
Code-7B | one-hop | 0.69±0.00 | 0.00±0.00 | 0.69±0.00 | 5.56±4.37 | 93.06±7.50 | 93.75 | 88.95
two-hop | 0.25±0.00 | 3.28±0.00 | 7.07±2.75 | 4.80±3.69 | 84.60±5.18 | 84.85
three-hop | 0.40±0.00 | 0.00±0.00 | 4.76±2.14 | 5.16±4.57 | 89.68±6.54 | 90.08
Code-13B | one-hop | 0.00±0.00 | 0.00±0.00 | 1.39±0.00 | 0.00±0.00 | 98.61±4.03 | 98.61 | 92.74
two-hop | 0.00±0.00 | 2.27±0.00 | 14.14±2.14 | 0.51±0.00 | 83.08±3.32 | 83.08
three-hop | 0.00±0.00 | 0.00±0.00 | 2.38±2.86 | 0.40±0.00 | 97.22±4.28 | 97.22

###  6.2 Model Alignment

We provide alignment strategy for both super large closed-source LLMs and
tunable open-source LLMs. (1) SoAyGPT is our proposed solution for models like
GPT Eloundou et al. (2023), which cannot have their parameters modified but
possess strong instruction following capabilities. We select a diverse set of
triplets from SoAyBench data that represent different usages of solutions as
examples to construct prompts. Given a query, SoAyGPT generates the solution
and code in sequence through in-context learning, followed by executing the
generated code to obtain the result. We utilize different versions of GPT
models as backbones for this purpose. (2) SoAyLLaMA, on the other hand, is our
alignment strategy for the LLaMA Touvron et al. (2023) model, which is an
open-source model that we can fine-tune within an acceptable training cost. By
using the query of the triplets from SoAyBench as input and integrating the
solution and code as output, SoAyLLaMA is trained to generate the solution and
code in a single step based on the given query in a zero-shot setting.

##  7 Experiments

In this section, we conduct experiments to validate the effectiveness,
efficiency, and practicality of SoAy. We focus on the following Research
Questions: (RQ1) In which part of the academic information seeking do LLMs
struggle the most? (RQ2) In terms of effectiveness on academic information
seeking, does SoAy outperform existing methods? (RQ3) In terms of efficiency,
does SoAy finish the task within an execution time that users can accept?
(RQ4) What kind of base model is more suitable for constructing SoAy?

We conduct experiments on SoAyBench to compare SoAy with other baseline
methods, focusing on these research questions. Additionally, we introduced a
detailed evaluation method called SoAyEval, which could also be used in the
evaluation of other tool-using scenarios. We also perform online evaluations
specifically targeting real-world academic information seeking use cases.
There’s an independent chapter in the appendix showing more implementation
details and interesting findings.

###  7.1 Experimental Settings

Baselines.  Many existing tool-utilizing works fail to function effectively
due to API inter-dependency and intricate execution logic. We have chosen GPT-
DFSDT, employing the same three versions of the GPT as backbone as SoAyGPT and
ToolLLaMA-7b Qin et al. (2023b) as baselines in API-using experiment on
SoAyBench, as the former represents the state-of-the-art model on a general
domain tool-utilizing benchmark, ToolBench Qin et al. (2023b), and the latter
is an open-source model that has been fine-tuned on numerous tool-utilizing
data samples, achieving results comparable to GPT-DFSDT. For every LLM, we
adopt a temperature setting of 0 to ensure reproducibility.

It is worth noting that the responses from the baselines, ToolLLaMa, and GPT-
DFSDT, are in natural language format, whereas the answers in SoAyBench are
the responses returned by SoAPI. In order to ensure a fair comparison, we have
designed a set of prompts to employ ChatGPT to extract precise answers from
responses in a natural language format.

SoAyEval.  Given a question that yields an API solution along with an invoked
answer, we have outlined five types of evaluation metrics. These metrics take
into account the accuracy of the solution, program, and answer together: (EM):
Both the retrieved solution and answer Exactly Match the ground truth. (DS):
The answer is correct, but a Different Solution is retrieved compared to the
ground truth. (WS): The answer is wrong due to a Wrong Solution. (WP): The
solution is correct but the answer is wrong, due to a Wrong Program generated
for the solution, which can be executed but yields the wrong answer. (EE):
Execution Error, which may by caused by the generation of a non-executable
program or network errors during the APIs request. Based on the aforementioned
metrics, we define the accuracy metric as follows:

$\displaystyle\text{ACC}=\text{EM}+\text{DS},$

which represents the proportion of correct answers produced by the method
under test. Finally, we assign different weights $w_{1}$, $w_{2}$, and $w_{3}$
to the ACC of one-hop, two-hop, and three-hop of the tested method to compute
the overall Score:

$\displaystyle\text{Score}=\frac{w_{1}\cdot\text{ACC}_{1}+w_{2}*\cdot\text{ACC}_{2}+w_{3}\cdot\text{ACC}_{3}}{w_{1}+w_{2}+w_{3}}.$

We name this evaluation method SoAyEval. SoAyEval meticulously analyzes the
entire process of large models using a set of APIs to answer corresponding
questions. It is a universal evaluation method that could be adopted by other
API-using scenarios. Notably, we assign $w_{1}=1,w_{2}=2,w_{3}=3$ in our
experiment to highlight the multi-hop question-answering ability of the
methods under test.

###  7.2 Overall Performance (RQ1, 2)

Table 3 presents the experimental results of SoAyGPT, SoAyLLaMA, and the other
two representative baseline methods on SoAyBench. SoAyLLaMA achieves the
highest overall score, with Code-13B Model of 91.24%, followed by the other
SoAy method, SoAyGPT, which gets 86.57% with GPT-4 as the backbone. These
results prove the effectiveness of SoAy. With the help of the prepared
solution library, SoAy describes the complex inter-dependency and diverse
execution logic to the LLMs. This significantly reduces the pressure of LLMs
when determining the optimal execution path from the coupling APIs.

Despite undergoing specialized tool usage training, ToolLLaMA does not score
highly. This is primarily because ToolLLaMA’s training objective is to master
the use of general APIs, where the dependencies involved are significantly
less complex than those in complicated academic APIs. Furthermore, many
questions in SoAyBench require a solution that incorporates branch or loop
logic. Facing with the need to execute these complicated execution logics,
GPT-DFSDT’s decision-tree-based inference encounters numerous repetitive sub-
nodes, making the solution exceptionally challenging.

###  7.3 Error Analysis (RQ1)

The detailed analysis of model performances from SoAyEval reveals distinct
characteristics and potential improvement areas for each model across
different question types. The SoAyLLaMA and SoAyGPT models, particularly their
latest versions, demonstrate superior capabilities in managing complex
question types with minimal errors. ToolLLaMA and early versions of GPT-DFSDT
show greater susceptibility to execution and solution pathway errors,
indicating potential areas for algorithmic or training enhancements. Overall,
advancements in model versions (notably from GPT-DFSDT 3.5 to 4 and SoAyGPT
3.5 to 4) show tangible improvements in reducing specific error types and
enhancing overall accuracy, highlighting the benefits of continuous model
development and refinement.

###  7.4 Efficiency (RQ3)

We evaluate the efficiency of the proposed method by reporting the average
response time of various methods evaluated on SoAyBench, as shown in Table 4.
We apply both SoAyLLaMA and ToolLLaMA in a network request service format to
keep justice with GPT-based methods which should only be accessed through APIs
calls. Both ToolLLaMa and GPT-DFSDT rely on multiple executions to complete
complex API operations, thus consuming a considerable amount of time.
Implementation through SoAy can effectively reduce the inference time of tool
calls. Even with GPT-4, the average execution time of SoAy is nearly one-third
of the execution time of GPT-DFSDT with GPT-4. Moreover, with a smaller
parameter scale, the lightweight SoAyLLaMA models show even less execution
time, which we consider already meet the requirement for being applied to an
academic searching service.

Table 4: Average response time of different methods (second).

Method | 7B | 13B | 3.5 | 3.5-16k | 4
---|---|---|---|---|---
ToolLLaMA | 45.10 | / | / | / | /
GPT-DFSDT | / | / | 39.12 | 53.73 | 70.92
SoAyGPT | / | / | 6.15 | 6.40 | 26.05
SoAyLLaMA-Code | 1.12 | 1.35 | / | / | /

###  7.5 Backbones Comparison (RQ4)

It’s worth considering which backbone model to choose to construct an academic
searching system whether through SoAyGPT or SoAyLLaMA. Based on the results of
SoAyBench, if we focus on the effectiveness of the results, the choice of
SoAyGPT definitely falls on the latest version (GPT-4 when the paper is
drafted). As for the choice of SoAyLLaMA, whose implementation is described in
Appendix B, we can find a trend where the scores increase as the number of
parameters grows. Simultaneously, given that Soay implements the planning and
formatting processes defined in the problem section through code generation,
CodeLlama, which excels in code generation, outperforms ChatLlama. It should
be mentioned that the effectiveness shown on the results of SoAyBench should
not be the sole criterion for judgment when considering practical
applications.

##  8 Related Works

APIs, short for Application Programming Interfaces, are the software
interfaces that provide services to other software
components111https://en.wikipedia.org/wiki/API#cite_note-1. It facilitates the
exchange of data between various software applications, thereby enabling
interoperability between them Ofoeda et al. (2019). For the application of
LLMs using the API-using methodology, the current solutions and technical
approaches can be divided into two major groups: A straightforward idea is to
broaden the tool/API-using coverage for LLMs. Such efforts not only include
basic web services, such as search engines Nakano et al. (2021); Liu et al.
(2023); Zhang et al. (2023) and calculators Schick et al. (2023), but also
include the intelligent multi-modal functions, such as vision understanding Wu
et al. (2023) and image generation Lu et al. (2023).

Others are devoted to the tool’s dependencies, programming logic and planning
ability King et al. (2023); Shen et al. (2023); Paranjape et al. (2023).
Except for the efforts that regard LLMs as database managers Qin et al.
(2023a), there are also plenty of methods that modeling the tool’s graphical
relations Zhang (2023), building intermediate instructions Burnett and
McIntyre (1995) and accomplish complex tasks Song et al. (2023).

Besides, applying LLMs to directly access the data from the Database through
techniques like text-to-SQL Qin et al. (2022) proves ineffective. The
limitation arises from the unavailability of access to the original data
sources of related systems.

##  9 Conclusion

In this work, we propose an API-Using methodology, SoAy, to apply LLMs to
academic information searching. The core technique of SoAy is the high-quality
automatic built dataset, SoAyBench, providing 3960 academic triplets data. We
align LLMs to generate codes on pre-constructed guidance, which could face the
coupling and execution logic diverse challenges. An evaluation strategy,
SoAyEval, is proposed to evaluate the AMiner API-using capability. Experiments
show SoAy’s effectiveness, efficiency, and usability. In the future, we will
continue to enlarge the SoAyBench and apply SoAy to more scenarios and
domains.

##  10 Limitations and Future Directions

In this paper, we propose an approach that combines automated data generation
with LLM alignment for academic query API-based tasks and demonstrates its
effectiveness. The released SoAyBench and the proposed SoAyEval could support
for evaluating related methods in subsequent studies. However, there are still
some worthwhile explorations that we have not conducted:

  * •

SoAyBench is a large-scale dataset constructed through automation, consisting
of 3960 triplet data instances. Although we incorporate quality checks in the
solution library construction, query generation, and code generation
processes, and have human experts inspect and refine the test set, we must
acknowledge that we cannot guarantee the provided tests in SoAyBench align
perfectly with people’s expectations for the API system. Some prior worksQin
et al. (2023b) validate the quality of their benchmarks by comparing the
results of tested methods on their own benchmark with manually evaluated
results. This is an aspect we plan to further explore and validate in the
future, which may lead to more human-aligned automated constructed datasets.

  * •

The API-using approach we propose addresses the task of guiding large models
to utilize complex academic search APIs. However, we have not yet validated
whether our method has advantages in simpler scenarios. We will continue to
expand SoAyBench and conduct practicality and effectiveness validations in
various scenarios.

  * •

In practical applications, we have encountered an inconvenient point of
updating the deployed system: if we want to incorporate new APIs on top of
existing ones, especially when adding a large number of APIs, the context-
guided SoAyGPT needs to adjust the structure of the prompt within a limited
input space. This poses a challenge that requires a trade-off between the
breadth and granularity of examples. Although our proposed training-based
SoAyLLaMA theoretically allows for the unlimited expansion of APIs by adding
sufficient training data, this is an aspect we have not explored yet. In the
future, we plan to explore methods for continuous learning with SoAyLLaMA to
enable rapid inclusion of new API data without compromising the model’s
mastery of existing APIs.

In conclusion, the quality of data generation, scalability, and the scenarios
presented in this paper opens up numerous possibilities for further
exploration. In the future, we will continue to fulfill the current work and
explore further in these mentioned directions.

##  11 Acknowledgements

This project would like to thank the data annotators at Zhipu AI for their
help and support. All GPU compute and API costs are provided by Zhipu AI.

##  12 Ethical Considerations

###  12.1 New datasets

Using SoAy, our new dataset is built on the AMiner dataset. The construction
of this new dataset relies on the efforts of annotators. We value their
contributions and provide them with reasonable compensation based on the
quantity of annotations and the duration of their work. For the
characteristics of the dataset, it focuses on the academic field and is
suitable for academic researchers. Furthermore, as talked before, to prevent
additional risks, we establish an evaluation benchmark, which is further
enhanced by additional human verification to ensure the quality of this new
dataset and its application effectiveness.

###  12.2 NLP application

The intended use of the SoAy trained LLMs is to facilitate researchers in
better accessing academic literature and other related resources. It allows
for querying in natural language, reducing the technical barriers for scholars
and others with academic information retrieval needs. However, just like the
issues inherent to LLMs, sometimes it may provide incorrect information,
potentially leading to misguidance for users, which may be a serious
disruption for researchers since academic information needs to be accurate.

## References

  * Burnett and McIntyre (1995) Margaret M Burnett and David W McIntyre. 1995.  Visual programming.  _COmputer-Los Alamitos-_ , 28:14–14.
  * dblp team (2005) The dblp team. 2005.  dblp computer science bibliography.
  * Eloundou et al. (2023) Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023.  Gpts are gpts: An early look at the labor market impact potential of large language models.  _arXiv preprint arXiv:2303.10130_.
  * Giles et al. (1998) C Lee Giles, Kurt D Bollacker, and Steve Lawrence. 1998.  Citeseer: An automatic citation indexing system.  In _Proceedings of the third ACM conference on Digital libraries_ , pages 89–98.
  * King et al. (2023) Evan King, Haoxiang Yu, Sangsu Lee, and Christine Julien. 2023.  " get ready for a party": Exploring smarter smart spaces with help from large language models.  _arXiv preprint arXiv:2303.14143_.
  * Liu et al. (2023) Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023.  Webglm: Towards an efficient web-enhanced question answering system with human preferences.  _arXiv preprint arXiv:2306.07906_.
  * Lu et al. (2023) Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023.  Chameleon: Plug-and-play compositional reasoning with large language models.  _arXiv preprint arXiv:2304.09842_.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021.  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_.
  * Noruzi (2005) Alireza Noruzi. 2005.  Google scholar: The new generation of citation indexes.
  * Ofoeda et al. (2019) Joshua Ofoeda, Richard Boateng, and John Effah. 2019.  Application programming interface (api) research: A review of the past to inform the future.  _International Journal of Enterprise Information Systems (IJEIS)_ , 15(3):76–95.
  * Paranjape et al. (2023) Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023.  Art: Automatic multi-step reasoning and tool-use for large language models.  _arXiv preprint arXiv:2303.09014_.
  * Qin et al. (2022) Bowen Qin, Binyuan Hui, Lihan Wang, Min Yang, Jinyang Li, Binhua Li, Ruiying Geng, Rongyu Cao, Jian Sun, Luo Si, Fei Huang, and Yongbin Li. 2022.  A survey on text-to-sql parsing: Concepts, methods, and future directions.  _Preprint_ , arXiv:2208.13629.
  * Qin et al. (2023a) Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. 2023a.  Tool learning with foundation models.  _arXiv preprint arXiv:2304.08354_.
  * Qin et al. (2023b) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023b.  Toolllm: Facilitating large language models to master 16000+ real-world apis.  _arXiv preprint arXiv:2307.16789_.
  * Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dess, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_.
  * Shen et al. (2023) Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023.  Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.  _arXiv preprint arXiv:2303.17580_.
  * Sinha et al. (2015) Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang. 2015.  An overview of microsoft academic service (mas) and applications.  In _Proceedings of the 24th international conference on world wide web_ , pages 243–246.
  * Song et al. (2023) Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. 2023.  Restgpt: Connecting large language models with real-world applications via restful apis.  _arXiv preprint arXiv:2306.06624_.
  * Tang et al. (2008) Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008.  Arnetminer: extraction and mining of academic social networks.  In _Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 990–998.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _Preprint_ , arXiv:2307.09288.
  * Wang et al. (2024) Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. 2024.  A survey on large language model based autonomous agents.  _Frontiers of Computer Science_ , 18(6).
  * Wu et al. (2023) Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023.  Visual chatgpt: Talking, drawing and editing with visual foundation models.  _arXiv preprint arXiv:2303.04671_.
  * Zhang (2023) Jiawei Zhang. 2023.  Graph-toolformer: To empower llms with graph reasoning ability via prompt augmented by chatgpt.  _arXiv preprint arXiv:2304.11116_.
  * Zhang et al. (2023) Jing Zhang, Xiaokang Zhang, Daniel Zhang-Li, Jifan Yu, Zijun Yao, Zeyao Ma, Yiqi Xu, Haohua Wang, Xiaohan Zhang, Nianyi Lin, et al. 2023.  Glm-dialog: Noise-tolerant pre-training for knowledge-grounded dialogue generation.  _arXiv preprint arXiv:2302.14401_.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_.

## Appendix

In the appendix, we’ll present in three folds the details of SoAy and
SoAyBench, along with more experimental details and conclusions. Part I focus
on the implementation of SoAy, including the details of SoAyGPT in A,
SoAyLLaMA in B and SoAyBench in C. Part II shows more about the experiments,
including the online evaluation in 4, the implementation of SoAy on other
academic information seeking systems in F, and more interesting findings from
the results on SoAyBench in E. Part III lists all the prompts used in SoAy and
SoAyGPT.

## Part I. Implementation Details

The content of this section primarily introduces some implementation details
of the SoAy method, including the inference architecture of SoAyGPT, the
training details of SoAyLLaMA, as well as the data details and API environment
setup details of SoAyBench. We implement SoAyGPT on three distinct versions of
GPT, specifically 3.5: gpt-3.5-turbo-0613; 3.5-16k: gpt-3.5-turbo-16k-0613,
and 4: gpt-4-0613. We conduct the fine-tuning on 3 different variants of the
LLaMA model, respectively Chat-7B: Llama-2-7b-chat-hf, Code-7B:
CodeLlama-7b-Instruct-hf, and Code-13B: CodeLlama-13b-Instruct-hf.

##  Appendix A SoAyGPT

SoAyGPT is an alignment method designed for a series of large-scale models
that are closed-source yet possess high command-following capabilities.
Specifically, SoAyGPT employs in-context learning to construct three LLM
agents Wang et al. (2024) (Solution Retrieval Agent, Code Generation Agent,
and Answer Generation Agent) and an environment operable by these agents to
complete the alignment process. The agents and environments are designed:

Solution Retrieval Agent.  Given the user input $Q$, the solution retrieval
agent obtains the appropriate solution from the solution library $S$. It’s
important to note that the retrieval process here diverges from traditional
information retrieval functions, which extract the exact required element from
a set. Essentially, this retrieval module presents all possible solutions as
the prompt to the LLMs, enabling them to generate a suitable solution.
Consequently, it’s possible that the produced solution differs from any
solution in $L_{S}$.

Code Generation Agent.  The code generation agent must generate the correct
program under the guidance of the retrieved solution $S$. Since the solution
might not be included in $L_{S}$, if the retrieved solution $S$ is in $L_{S}$,
we simply use $P_{p}$ specifically defined for $S$ to generate the required
program. Otherwise, SoAy has to use $P_{g}$ that is defined for all the
possible solutions to guide the LLMs in generating a program, as no specific
solution could be identified. SoAy employs Python as the programming language
for each program, allowing it to incorporate not only sequential but also
branching and looping logic.

Code Execution Environment.  SoAy employs a virtual Python environment as the
executor to obtain the results, serving as the knowledge $K$. Here, $K$ is
expected to be the exact answer to $Q$ if $Q$ asks about some precise
information, like the number of citations for a scholar. Otherwise, $K$
provides helpful information if $Q$ has an ambiguous intent.

Answer Generation Agent.  Finally, SoAy generates a natural language answer
$A$ based on $Q$ and $K$, guided by $P_{a}$.

##  Appendix B SoAyLLaMA

SoAyLLaMA is designed for supervised fine-tuning of open-source models when
training resources are available. Through the training stage, SoAyLLaMA
integrates an API description and queries as inputs and combines solutions and
programs as responses. During the inference stage, SoAyLLaMA concurrently
completes the Planning and Formatting processes to derive solutions and code.
Similar to SoAyGPT, it utilizes the environment to execute and obtain answers
based on $K$. We conduct the fine-tuning on 3 different variants of LLaMA
Model, respectively Chat-7B: Llama-2-7b-chat-hf222https://huggingface.co/meta-
llama/Llama-2-7b-chat-hf, Code-7B: CodeLlama-7b-Instruct-
hf333https://www.modelscope.cn/models/AI-ModelScope/CodeLlama-7b-hf/summary,
Code-13B: CodeLlama-13b-Instruct-hf444https://huggingface.co/meta-
llama/CodeLlama-13b-Instruct-hf.

Dataset.  As mentioned in Section 3, four-fifth of the instances from
SoAyBench are utilized here to train the SoAyLLaMA. We simply take the
academic query as an input and the relevant solution.

Training Details.  We train 13B models on 8 * NVIDIA A100 GPU (80GB) for 13
hours to accomplish a convergent loss, with a batch size of 8. 7B models are
trained on 4 * NVIDIA A100 GPU (80GB) for 7 hour with a batch size of 16. All
the training details are saved in the training log in our Github
Repo555https://anonymous.4open.science/r/SoAy-C025/readme.md.

##  Appendix C SoAyBench

To assess AMiner API-using capabilities, the foundational AMiner APIs for LLMs
to invoke and a test set should be provided. However, given the dynamic and
specialization nature of academic data, with academic entities-related
information rapidly changing, maintaining a test set with static answers
proves challenging. To address this challenge, we clone AMiner’s APIs at a
specific point in time to create a static service, from which we create a
corresponding static test set including 792 meaningful academic questions
(one-fifth of the whole SoAyBench) and ground truth conducted from the cloned
APIs. We extract a subset of the comprehensive AMiner database, which includes
over 66,000 entities such as persons and publications, at a specific point in
time, September 23, 2023. This subset serves as our experimental evaluation
environment, and we clone an API Service based on it. Consequently, the
results returned by these APIs, given the same parameters, will remain
consistent over time. This dataset and the API service provide a foundation
for our subsequent construction of the test set.

## Part II. Experiments Details

In this section, we will present more details related to the experiments. To
verify the efficiency and effectiveness of SoAy in real applications, it is
insufficient to rely solely on its performance on SoAyBench, so we designed an
online evaluation to test SoAy’s performance in real-use scenarios. Beyond the
research questions in Section 7, more interesting insightful conclusions can
be found in the results on SoAyBench. Furthermore, to validate the
universality of the SoAy method, rather than it only working for the AMiner
system, we implemented SoAyGPT in two other academic information seeking
scenarios and demonstrated its results in a case study manner.

##  Appendix D Online Evaluation

Figure 4: Distribution of questions used for online human evaluation.
Questions are categorized by 1, 2, and 3-hop types, further divided into exact
and ambiguous categories. The ambiguous category is specific to this online
evaluation setting.

We examine the Spearman correlation coefficients among the voting results of
the 15 human annotators, as shown in Figure 5. The released SoAyBench contains
the templated questions derived from the AMiner SoAPI, thus testing a method’s
ability to understand and utilize the SoAPIs to a certain extent. However, in
real-world deployment scenarios, the goal of a good method should be to
fulfill user needs more comprehensively and offer a superior user experience.
To achieve this, we implement the proposed SoAy as an online application,
gather 52 real user demands from the logs, and invite 10 annotators to conduct
human evaluation.

Figure 5: Spearman correlation among 15 annotators

Settings.  We deploy SoAy with GPT-3.5-16k as its backbone online, due to the
superior speed of the 3.5 version, enhancing user experience. Over time, we
gather 52 authentic questions from the application logs. The distribution of
these questions is illustrated in Figure 4, categorized not only by their 1,
2, and 3-hop nature but also classified as either exact or ambiguous.
Ambiguous questions, such as requests for “comparisons between two scholars”,
lack standardized answers and thus yield diverse responses. We find that the
majority of these 52 questions could be resolved within two hops.

To evaluate the answers, we invite five academic experts to provide responses
based on their expertise, supplemented by the AMiner system and general search
engines. The online system’s answers, responses directly from ChatGPT, and the
aforementioned human-provided answers constitute the comparison methods.

For a balanced and fair assessment of the experimental results, we develop an
annotation platform and create a detailed annotation manual. This platform
allows annotators to shuffle and anonymize the answers from the three methods
before each voting session. A team of 10 annotators is recruited to assess the
answers provided by each method for every question across a two-day period.
Their focus is on evaluating the accuracy and readability of the responses,
culminating in a vote for the most effective response among the three. We also
open-source the platform and its associated manual.

Evaluation Results.

Figure 6: Voting results for authentic online questions.

After identifying the three annotators with the lowest correlation to others,
we exclude their input. For the remaining annotators, a majority voting
strategy is employed to consolidate the votes for the three methods. The
summarized results are presented in Figure 6. These results show a preference
for the SoAy method’s responses in real-world scenarios across various
question types, outperforming both direct ChatGPT and human expert responses.

Notably, SoAy’s responses are particularly favored for questions with exact
answers. However, SoAy’s advantage is less pronounced for ambiguous questions
when compared to baseline responses. This observation aligns with SoAy’s
design and the assessment objectives of SoAyBench, both of which prioritize
precise answering through the integration of SoAPI Library resources.
Nevertheless, there’s a noticeable gap between the nature of real-world
inquiries and the templated questions in SoAyBench, posing a challenge for
SoAy in addressing certain scenarios. This finding points towards the
potential for integrating SoAPI calls with model-intrinsic functions for a
more robust response system.

##  Appendix E Insights on SoAyBench

###  E.1 Intriguing phenomena

Table 3 reveals several other intriguing phenomena: First, while SoAy excels
with GPT-4, its advantages are less marked on GPT-3.5-turbo and GPT-3.5-16k
(hereafter collectively referred to as GPT-3.5 backbones), trailing behind the
GPT-DFSDT method with the same GPT-3.5 backbones. Notably, the DS and WS
metrics for SoAy with GPT-3.5 backbones are quite high. This can be attributed
to GPT-4’s superior instruction-following proficiency, which amplifies the
impact of SoAy’s complex prompts on the model adept at instruction following.
Second, SoAy methods exhibit superior performance on the two-hop questions
than one and three-hop questions, particularly with GPT-3.5 backbones. This is
likely because the solution parsing prompts sampled from the SoAyBench are
biased towards two-hop questions (refer to Table 2 for more information),
causing the GPT-3.5 backbones to default to two-hop solutions within the SoAy
framework. On the contrary, GPT-4’s robust instruction-following ability
enables it to perform consistently across all question types. Third, GPT-DFSDT
exhibits a stronger understanding of one-hop questions when using GPT-3.5
backbones. It breaks down complex questions into simpler one-hop questions,
determining the appropriate APIs and their parameters step by step, which
enhances its comprehension of one-hop questions.

###  E.2 Variances

Notably, the data fluctuations marked in Table 3 do not represent variances
represented by repeating the identical experiment on identical samples. Given
that the $768$ test samples in SoAyBench originate from $44$ template
questions, and each template is filled with different entities to create
unique question instances, we compute each metric within a group of
instantiated questions from the $44$ question templates. We then calculate the
mean and variance across groups of different instantiated questions. Our
observations indicate that SoAyGPT-4 and SoAyLLaMA delivers a more
consistently stable performance across various error types, which benefits
from GPT-4’s enhanced generalization capacity across a variety of entities and
sufficient and diverse training data from SoAyBench.

###  E.3 Detailed Error Analysis

Notably, We do not rectify network errors because SoAy aims to apply the
system in real-world use, implying that it is essential to test each model’s
stability. During experiments, we attempt to request the models up to 3 times.
If all 3 attempts fail, the situation is classified as an EE.

####  E.3.1 ToolLLaMA (7B)

Shows the highest Execution Error (EE) rates across all question types, which
might indicate issues with the model’s ability to generate executable programs
consistently. It struggles particularly with one-hop questions, which have the
lowest Exact Match (EM) rate and high Wrong Solution (WS) errors. Two-hop and
three-hop questions reflect similar challenges with WS and EE, although DS
errors decrease slightly as the complexity increases. High variability in
error rates suggests that the model might benefit from training focused on
error reduction techniques. The overall score of 16.72 is the lowest among the
models, reflecting significant room for improvement.

####  E.3.2 GPT-DFSDT

Version 3.5:  One-hop questions see extremely high DS rates but lower WS and
EE, suggesting issues with solution path diversity but not necessarily with
execution. Two-hop and three-hop configurations show a balance of errors
across DS, WS, and EE, with a slightly better EM rate for three-hop.

Version 3.5-16K:  This model shows improved performance over the standard 3.5
version, with particularly notable improvements in EM and ACC for one-hop
questions. DS errors are significantly reduced compared to the 3.5 version,
indicating enhanced solution alignment.

Version 4:  The most advanced version, showing the best performance with
notably high scores, especially in one-hop questions which almost reach
perfect EM and ACC. Lower WS rates across all configurations, indicating
robustness in generating correct solutions.

####  E.3.3 SoAyGPT

Version 3.5:  Generally strong in EM and ACC, particularly for one-hop
questions, suggesting effective solution generation and execution. Lower EE
than ToolLLaMA, indicating more reliable program execution.

Version 3.5-16K:  Consistently high EM rates across question types, with
particularly low DS errors in two-hop questions. Shows good balance across all
error types, with no extreme weaknesses.

Version 4:  Outstanding performance in one-hop questions with nearly no errors
and very high EM and ACC. Two-hop and three-hop questions also show very high
scores, making this version highly effective and reliable.

####  E.3.4 SoAyLLaMA

Chat-7B:  Exhibits nearly perfect performance in one-hop questions with zero
DS, WS, and WP errors. Three-hop questions maintain high performance but see a
slight increase in WP errors, which still remain low compared to other models.

Code-7B:  Solid performance with minimal error rates across all question
types. A slight uptick in WP errors in two-hop questions could be a focal
point for model refinement.

Code-13B:  The best-performing model across all metrics, with impressive EM
and ACC in one-hop questions and consistent strength in multi-hop
configurations. Nearly negligible error rates, setting a high standard for
large language models.

##  Appendix F Other Scenarios

###  F.1 SoAy implemented on CrossRef

####  F.1.1 Introduction

Crossref is a registration agency for DOIs (Digital Object Identifiers)
established by a consortium of renowned global publishers. It registers DOI
numbers for newly published and previously published journal articles. All
works registered with Crossref by various publishers automatically establish
citation links with each other, facilitating readers to navigate journals and
access the websites of different publishers, thus significantly increasing the
chances of journal articles being accessed and cited. Crossref’s open
reference linking system establishes citation links to journal articles
between different publisher websites. For developers, Crossref provides rich
open-source API interfaces on GitHub666https://github.com/CrossRef/rest-api-
doc?tab=readme-ov-file, such as search and other functionalities.

####  F.1.2 Implementation

The codes, prompt, and APIs of this part of the implementation are available
at our GitHub repo:
https://github.com/RUCKBReasoning/SoAy/tree/master/scenarios/CrossRef. As
shown above, we need to finish two primary stages: the offline solution
library preparation stage and the online solution-based inference stage.

Figure 7: Illustration of Crossref SoAPI Library, SoAPI Graph and solution
library.

API Library.  As shown in Figure 7, we use 6 APIs of Crossref. The phenomenon
of API coupling is prevalent among these SoAPIs. For example, the parameter
doi for getWorksByDoi(·) must first be obtained by searchWorkByTItle(·). This
necessitates a specific execution order where searchWorkByTItle(·) must be
used before getWorksByDoi(·), and a specific execution composition where if
getWorksByDoi(·) is used, searchWorkByTItle(·) is definitely used.

Solution Library Construction.  The solution library is the cornerstone of the
entire framework. Just as what we did for AMiner, we finish the solution
library for Crossref, proving SoAy’s usability. The initial stage of SoAy
involves analyzing Crossref’s SoAPI and constructing the SoAPI Graph.
Following this, a process of sampling and filtering paths is applied to the
SoAPI Graph, resulting in the solution library. Based on it, the prompt
library is then constructed.

To better understand the performance of Soay on Crossref, let’s take a
detailed look at Crossref’s prompts and cases on Figure 10

General Prompt Drafting.  The General Prompt outlines a structured approach to
crafting programs that utilize a set of APIs to address various queries
related to literature and authors. It starts by presenting the APIs that will
be employed, such as ‘searchBook‘, ‘getBook‘, and others, which are
instrumental in searching for books, obtaining book details, and accessing
author information. The prompt then proceeds to illustrate the process through
a series of examples. The first example guides the user on how to use the
‘searchBook‘ API to find publishers in a particular subject area (XXX),
iterating through the results to compile a list of publisher names. This
demonstrates the initial step of narrowing down the scope of the query to a
more manageable list. Following that, the second example builds upon the first
by not only identifying publishers in a specific field but also by delving
deeper to list articles or works associated with those publishers. This is
achieved by chaining the ‘searchPublisherBySubject‘ API to
‘getPublisherBasicInfo‘ and then to ‘getPublisherWorks‘, showcasing a more
complex use of the APIs to extract detailed information. The prompt concludes
with a user query placeholder, encouraging the LLMs to apply the learned
methods to formulate a solution. This involves selecting the correct sequence
of APIs to retrieve the necessary information in response to the query. The
prompt is designed to be a comprehensive guide that adapts to different levels
of complexity in the user’s requests, from simple to more advanced searches.

####  F.1.3 Case Study

Case 1: Mathematics Publishers. The first case presents a query for a list of
publishers in the mathematics field. The solution uses the
‘searchPublisherBySubject‘ API with ’math’ as the input to extract a list of
publishers, along with the number of DOIs they’ve published. The result
provides a collection of publisher names and their respective DOI counts,
highlighting their impact in the field of math.

Case 2: DOI Numbers in Computer Science. The second case addresses the request
for current DOI numbers from publishers in computer science. By utilizing
‘searchPublisherBySubject‘ to identify relevant publishers, followed by
‘getPublisherBasicInfo‘ to obtain DOI counts, the solution compiles a list of
publishers with their current DOI numbers, showcasing their recent publication
activities.

Case 3: Article Titles in Computer Science. The third case involves listing
articles in the computer science domain. It starts with
‘searchPublisherBySubject‘ to find publishers, then uses
‘getPublisherBasicInfo‘ and ‘getPublisherWorks‘ to retrieve a list of article
titles. The output is a concise list of article titles that represent the
range of research within computer science.

###  F.2 SoAy implemented on OpenLibrary

####  F.2.1 Introduction

Open Library is an online project aimed at creating a comprehensive database
of all published books. Operated by the Internet Archive, the website has been
serving the public since 2006. It provides information pages for thousands of
books, displaying their summaries, covers, publication details, and more.
Additionally, it features personal pages for numerous authors, detailing their
biographical information, major works, and areas of expertise. For developers,
the website also offers web APIs such as search functionality, which are
documented in the Public APIs project.

####  F.2.2 Implemention

The codes, prompt, and APIs of this part of the implementation are also
available at our GitHub repo:
https://github.com/RUCKBReasoning/SoAy/tree/master/scenarios/OpenLibrary. The
same as AMiner and CrossRef, we need to finish two primary stages: the offline
solution library preparation stage and the online solution-based inference
stage.

Figure 8: Illustration of OpenLibrary SoAPI Library, SoAPI Graph and solution
library.

API Library.  As shown in Figure 8, we use 6 APIs of Crossref. The phenomenon
of API coupling is prevalent among these SoAPIs. For example, the parameter
subject for searchSubject(·) must first be obtained by getBook(·) or
getAuthorBasicInfo(·). This necessitates a specific execution order where
getBook(·) or getAuthorBasicInfo(·) must be used before searchSubject(·), and
a specific execution composition where if searchSubject(·) is used, getBook(·)
or getAuthorBasicInfo(·) is definitely used.

Solution Library.  The solution library is the cornerstone of the entire
framework. Just as what we did for AMiner, we finish the solution library for
OpenLibrary, proving SoAy’s usability. The initial stage of SoAy involves
analyzing OpenLibrary’s SoAPI and constructing the SoAPI Graph. Following
this, a process of sampling and filtering paths is applied to the SoAPI Graph,
resulting in the solution library. Based on it, the prompt library is then
constructed.

General Prompt.  To better understand the performance of Soay on OpenLibrary,
let’s take a detailed look at OpenLibrary’s prompts and cases on Figure 11 It
starts by presenting the APIs that will be employed, such as ‘searchBook‘,
‘getBook‘, and others, which are instrumental in searching for books,
obtaining book details, and accessing author information. The prompt then
proceeds to illustrate the process through a series of examples. The first
example guides the user on how to use the ‘searchBook‘ API to find publishers
in a particular subject area (XXX), iterating through the results to compile a
list of publisher names. This demonstrates the initial step of narrowing down
the scope of the query to a more manageable list. The same principle applies
to the two other examples. The prompt concludes with a user query placeholder,
encouraging the LLMs to apply the learned methods to formulate a solution.
This involves selecting the correct sequence of APIs to retrieve the necessary
information in response to the query. The prompt is designed to be a
comprehensive guide that adapts to different levels of complexity in the
user’s requests, from simple to more advanced searches.

####  F.2.3 Case Study

Case 1: Identifying Mathematics Publishers. In the first case, the objective
is to identify publishers that specialize in mathematics. The process begins
by utilizing the ‘searchPublisherBySubject‘ API, specifying ’math’ as the
subject parameter. This call returns a list of publishers active in the field
of mathematics, each accompanied by the number of DOIs (Digital Object
Identifiers) they have generated. The output is a curated list of publisher
names alongside their DOI counts, which serves as an indicator of their
contributions and presence in the mathematical literature space.

Case 2: Obtaining DOI Numbers for Computer Science Publishers. The second case
focuses on obtaining the current DOI numbers for publishers within the
computer science field. The ‘searchPublisherBySubject‘ API is employed to
first filter out publishers related to computer science. Once a list of these
publishers is obtained, the ‘getPublisherBasicInfo‘ API is then used to fetch
the up-to-date DOI numbers for each publisher. The result is a compilation of
publisher names paired with their current DOI counts, providing a snapshot of
their recent scholarly output and engagement within the field.

Case 3: Listing Computer Science Articles. The third case aims to extract a
list of article titles from the computer science domain. The process initiates
with the ‘searchPublisherBySubject‘ API to pinpoint publishers in the field,
followed by the use of ‘getPublisherBasicInfo‘ to acquire the publisher’s
unique ID. With the publisher ID in hand, the ‘getPublisherWorks‘ API is then
applied to retrieve a comprehensive list of article titles associated with the
publisher. The final output is a succinct array of article titles, offering a
glimpse into the diverse range of research topics and findings published
within the realm of computer science.

## Part III. Prompts

In this section, we will comprehensively present the prompts used in the data
generation process of SoAy, as well as the implementation process of SoAyGPT
on AMiner and two other scenarios.

Figure 9: A Preview of the prompt library. $P_{s}$ signifies the prompt for
solution retrieval, $P_{p}$ represents the prompt for program generation, and
$P_{a}$ denotes the prompt for answer generation. All the black text except
the ellipsis is the original prompt content, the yellow highlighted text is
the annotations added for better understanding in the context of creating a
prompt, and the purple highlighted text is the content waiting for the LLM
output. Figure 10: A Preview of the prompt library and three cases for
Crossref.  Figure 11: A Preview of the prompt library and three cases for
OpenLibrary.  Figure 12: A Preview of the prompts used in the data preparation
process.  Figure 13: The screenshots of the demonstration systems
https://soay.aminer.cn and online service widget
https://chatglm.cn/main/gdetail/65bf5a99396389a73ace9352 powered by SoAy.