# Granite-Function Calling Model: Introducing Function Calling Abilities via
Multi-task Learning of Granular Tasks

###### Abstract

Large language models (LLMs) have recently shown tremendous promise in serving
as the backbone to agentic systems, as demonstrated by their performance in
multi-faceted, challenging benchmarks like SWE-Bench and Agent-Bench. However,
to realize the true potential of LLMs as autonomous agents, they must learn to
identify, call, and interact with external tools and application program
interfaces (APIs) to complete complex tasks. These tasks together are termed
function calling. Endowing LLMs with function calling abilities leads to a
myriad of advantages, such as access to current and domain-specific
information in databases and knowledge sources, and the ability to outsource
tasks that can be reliably performed by tools, e.g., a Python interpreter or
calculator. While there has been significant progress in function calling with
LLMs, there is still a dearth of open models that perform on par with
proprietary LLMs like GPT, Claude, and Gemini. Therefore, in this work, we
introduce the Granite-20B-FunctionCalling111The model will be available soon
at https://huggingface.co/ibm-granite/ model under an Apache 2.0 license. The
model is trained using a multi-task training approach on seven fundamental
tasks encompassed in function calling, those being Nested Function Calling,
Function Chaining, Parallel Functions, Function Name Detection, Parameter-
Value Pair Detection, Next-Best Function, and Response Generation. We present
a comprehensive evaluation on multiple out-of-domain datasets comparing
Granite-20B-FunctionCalling to more than 15 other best proprietary and open
models. Granite-20B-FunctionCalling provides the best performance among all
open models on the Berkeley Function Calling Leaderboard and fourth overall.
As a result of the diverse tasks and datasets used for training our model, we
show that Granite-20B-FunctionCalling has better generalizability on multiple
tasks in seven different evaluation datasets.

Ibrahim Abdelaziz⋆† Kinjal Basu⋆† Mayank Agarwal⋆† Sadhana Kumaravel Matthew
Stallone Rameswar Panda Yara Rizk GP Bhargav Maxwell Crouse Chulaka Gunasekara
Shajith Ikbal Sachin Joshi Hima Karanam Vineet Kumar Asim Munawar Sumit Neelam
Dinesh Raghu Udit Sharma Adriana Meza Soria Dheeraj Sreedhar Praveen
Venkateswaran Merve Unuvar David Cox Salim Roukos Luis Lastras Pavan
Kapanipathi†

IBM Research

⋆Equal Contribution

†Corresponding Authors

{ibrahim.abdelaziz1, kinjal.basu<EMAIL_ADDRESS><EMAIL_ADDRESS>
##  1 Introduction

Large language models (LLMs) have garnered significant attention due to their
broad applicability to an important set of challenging domains, e.g.,
programming (Mishra et al., 2024; Roziere et al., 2023), reasoning (Reid et
al., 2024; Jiang et al., 2023), and multi-modal interaction (Reid et al.,
2024). Increasingly, applying these models to solve real-world problems
requires them to act as autonomous agents powering intelligent decision-making
in specific environments (Yao et al., 2022; Xu et al., 2023a; Yang et al.,
2024a)222Auto-GPT:https://github.com/Significant-
Gravitas/AutoGPT333BabyAGI:https://github.com/yoheinakajima/babyagi. For LLMs
to serve as autonomous agents, they must perform accurately on two fundamental
capabilities: (a) reasoning and planning, and (b) function calling, which
includes identifying, calling, and interacting with tools and APIs in external
environments. In this work, we focus on improving LLMs’ function calling
abilities.

Function calling provides a means for language models to leverage external
tools and resources. These tools can make available to an LLM specific, up-to-
date information that would otherwise be inaccessible (e.g., stored in a
dynamic knowledge base) and thus reduce its proclivity for hallucinating
responses (Schick et al., 2023). This is particularly crucial in enterprise
use cases where a significant portion of relevant data is stored in a
structured format accessible only via storage engines. In addition to
knowledge access, function calling can allow an LLM to outsource tasks that
are out of scope for a generalized language model. Most commonly, these tasks
involve compute-heavy operations, e.g., program execution (Shinn et al.,
2023), numerical calculation, or retrieval (Schick et al., 2023), and are
otherwise a frequent source of LLM hallucinations (Li et al., 2023a).

The importance of function calling has spurred the development of several
recent data generation efforts for fine-tuning (Basu et al., 2024; Guo et al.,
2024; Qin et al., 2023; Yan et al., 2024; Tang et al., 2023) and evaluation of
models (Li et al., 2023b; Muennighoff et al., 2023). Typically, however, the
fine-tuned models from datasets like ToolLLM (Qin et al., 2023), ToolAlpaca
(Tang et al., 2023), and Gorilla (Patil et al., 2023) underperform in one (or
more) of three key dimensions: (a) Generalizability: While the datasets are
generated using diverse sets of APIs (e.g., ToolLLama uses RapidAPIs
444https://rapidapi.com/hub, ToolAlpaca uses public
APIs555https://github.com/public-apis/public-apis, and Gorilla uses TensorFlow
Hub, PyTorch Hub, and Hugging Face Hub), work from (Basu et al., 2024) has
shown that models trained on these datasets have difficulty generalizing to
out-of-domain datasets. (b) Granular tasks: Function calling, as an umbrella
term, can encompass multiple granular sub-tasks such as function-name
detection, slot filling666Slot, parameter, and argument are used
interchangeably. or parameter-value pair detection, and detecting the ordered
sequence of functions needed to be called. Existing models trained to perform
function calling lack the ability to handle these granular tasks
independently, and hence, perform poorly on such sub-tasks. (c) Openness: The
best performing models are proprietary and the ones that have open licenses
(e.g., Gorilla (Patil et al., 2023)) are trained using data generated from
OpenAI models.

Figure 1: Step-by-step building process of Granite-20B-FunctionCalling

To address the aforementioned limitations, in this work, we focus on
introducing function-calling abilities to models with an inherent focus on
granular tasks. Figure 1 shows an overview of how Granite-20B-FunctionCalling
was trained. The datasets used for training are API-Blend (Basu et al., 2024)
that include tasks such as function name detection, slot filling, parallel
functions, multiple functions, sequencing777Sequencing and chaining are used
interchangeably., and calling APIs888 Function and API are used
interchangeably. using multiple programming languages. We build upon Granite
code models by instruction tuning them for function calling using the datasets
for granular tasks with a multi-task learning approach. Granite code models
are trained on data that is license permissible following IBM’s AI Ethics
principles for trustworthy enterprise usage (Mishra et al., 2024). Being part
of the Granite family, we release Granite-20B-FunctionCalling under Apache 2.0
license. Finally, in this work, we perform a comprehensive evaluation of the
open and proprietary models using Berkeley Function Calling Leaderboard
(BFCL), four Function Calling Academic Benchmarks, and Response Generation
Benchmark from API-Bank (Li et al., 2023b) to evaluate the generalizability of
function-calling models. Granite-20B-FunctionCalling is on par with the best
open model on BFCL and fourth overall. Furthermore, compared to other models
based on the out-of-domain datasets, Granite-20B-FunctionCalling shows
significant generalizability. Figure 2 shows how Granite-20B-FunctionCalling
compares to the top two open models (according to BFCL) on various tasks where
despite only having 20B parameters, it performs as well or better than Meta-
Llama-3-70B-Instruct which has 70B parameters.

Figure 2: Evaluation of Granite-20B-FunctionCalling against the best open
function calling models (according to BFCL)

##  2 Related Work

###  2.1 Instruction Tuning

Our work is an instantiation of instruction tuning (Wei et al., 2021), a fine-
tuning method that improves an LLM’s ability to solve natural language tasks
(Mishra et al., 2022; Wang et al., 2023). It involves taking a large
collection of NLP datasets, reformulating those datasets into a set of
instruction-following tasks, and then fine-tuning an LLM on the modified data.
While the earliest versions of instruction tuning straightforwardly combined
large datasets together, the most recent iterations use more sophisticated
mixtures of tasks to achieve the best results (Li et al., 2024; Sudalairaj et
al., 2024). Our work draws largely upon API-Blend (Basu et al., 2024) and API
Pack (Guo et al., 2024), two recently introduced instruction-tuning datasets
specifically focused on tasks related to APIs, e.g., slot filling and API
intent detection.

Instruction-tuned models often significantly outperform their base models on a
wide range of tasks, particularly in the zero-shot setting (Ouyang et al.,
2022; Muennighoff et al., 2023; Chung et al., 2022). Further improvement has
been observed through alignment of the model after instruction-fine-tuning,
e.g., TP-LLaMA (Chen et al., 2024a) uses Direct Preference Optimization
(Rafailov et al., 2023) in addition to fine-tuning.

There is also growing interest in developing smaller models that match or
surpass the accuracy of larger proprietary LLMs in function calling tasks
(Chen et al., 2024b). These compact models are crucial in the increasing
prevalence of on-device LLMs, enabling efficient and effective performance on
local devices.

###  2.2 Function Calling by LLMs

Function calling augmentation has broadened the scope of problems addressable
by LLMs to include those that cannot be solved with internal knowledge alone.
For instance, prior work has demonstrated the use of API-enhanced LLMs to
solve problems requiring up-to-date information retrieval (Schick et al.,
2023), intricate mathematical calculations (He-Yueya et al., 2023; Patel et
al., 2021), internet use (Komeili et al., 2022; Gur et al., 2023), task
orchestration (Jain et al., 2024), and even programming abilities (Gao et al.,
2023).

Multiple strategies have been proposed for how best to enable LLM function
calling. One line of prior research has investigated the design of elaborate
prompting approaches, best exemplified by the popular ReACT prompting
framework (Yao et al., 2022). Such prompting methods can vary in their design,
with some works optimizing for cost (Xu et al., 2023a), raw performance (Shinn
et al., 2023; Yang et al., 2023), or a blend of both (Crouse et al., 2023).
More relevant to our approach are methods that train models to directly output
function calls (Tang et al., 2023; Qin et al., 2023). Typically, these works
will use some form of self-supervision to enable scaling to the breadth of
domains required for general-purpose function use (Schick et al., 2023; Parisi
et al., 2022; Yang et al., 2024b).

Recently, many language models with function-calling capabilities have been
introduced. They broadly fall into two categories: pre-trained models which
are capable of function-calling (Reid et al., 2024; CodeGemma Team et al.,
2024; CohereForAI, 2024; AI@Meta, 2024; Jiang et al., 2023), and models fine-
tuned specifically for function-calling (Qin et al., 2023; Tang et al., 2023;
MeetKai, 2024; Patil et al., 2023; Nous-Research, 2023; Nexusflow.ai, 2023).
While the pre-trained models enable function-calling using a combination of
supervised and preference fine-tuning, details of the datasets used to train
models for these tasks are not generally available. On the other hand,
specialized function-calling models mostly rely on synthetic data generated
from proprietary state-of-the-art models. Models like Gorilla (Patil et al.,
2023), ToolLlama (Qin et al., 2023), ToolAlpaca (Tang et al., 2023), and the
NousResearch Hermes series of models (Nous-Research, 2023) utilize GPT-4 or
ChatGPT to generate synthetic instruction tuning datasets and fine-tune a base
model such as the Llama or Mistral model for function-calling tasks. The
NexusRaven models (Nexusflow.ai, 2023) are one of the few open-source models
that focus on building function-calling models for commercial purposes by
avoiding using proprietary models for synthetic data generation.

In section 5, we compare our model to the above models and show that
Granite-20B-FunctionCalling provides the best or comparable performance
amongst all open models across multiple tasks.

##  3 Multi-Task Training Data

In this section, we describe our detailed approach to fine-tune the
Granite-20B-Code-Instruct model with multi-task data related to functions to
build Granite-20B-FunctionCalling, a robust model designed for function-
calling. We use API-BLEND (Basu et al., 2024), a diverse corpora of multiple
API datasets for training LLMs. It consists of five datasets with a total of
about 160K training examples: SeqSGD, SeqSNIPS, SeqTopV2, SeqATIS, and
SeqMultiWOZ.

A key contribution to the process of building Granite-20B-FunctionCalling is
multi-task training, where we reuse the same data in different formats with
distinct instructions for different function-calling related tasks. We have
identified six underlying sub-tasks for function calling and divided them into
two broad categories based on the difficulty levels: (A) High-Level Function
Calling Tasks which are complex tasks for an LLM and typically handle multiple
functions; and (B) Low-Level Function Calling Tasks which are simpler tasks
for an LLM and relate to either function names or only parameter-value pairs.
We have included “Response Generation” as the seventh task in our training
data since producing natural language responses is one of the fundamental
goals of an LLM. Table 1 demonstrates the task-wise mapping of each dataset.
Below, we describe each task in detail.

| High-Level Function Calling Tasks | Low-Level Function Calling Tasks |
---|---|---|---
Datasets |  Nested Func. Calling |  Func. Chaining |  Parallel Func. |  Next-Best Func. |  Func. Name Detection |  Param-Val Pair Detection |  Response Generation
SeqSGD |  | ✔ | ✔ | ✔ | ✔ | ✔ |
SeqSNIPS |  | ✔ | ✔ | ✔ | ✔ | ✔ |
SeqTopV2 | ✔ | ✔ |  | ✔ | ✔ | ✔ |
SeqATIS |  | ✔ | ✔ | ✔ | ✔ | ✔ |
SeqMultiWOZ |  | ✔ |  | ✔ | ✔ | ✔ |
Glaive-V2 |  | ✔ |  |  |  |  | ✔
Table 1: Training Datasets with Task mapping

In the rest of the section, we describe how to unify the data of different
datasets in the same format for model training and then describe each of these
training tasks.

###  3.1 Data Unification

In addition to the datasets in API-BLEND (Basu et al., 2024) (i.e., SeqSGD,
SeqSNIPS, SeqTopV2, SeqATIS, SeqMultiWOZ), we also use
Glaive-V2999https://huggingface.co/datasets/glaiveai/glaive-function-
calling-v2 to prepare the training data mixture for
Granite-20B-FunctionCalling, where each dataset is multi-purposed for
different function calling related tasks with different instructions. These
datasets come from different sources and have various function formats that
require unification to an identical representation for better usability.

In this unification process, we convert all the APIs, tools, and functions
from the data into a JSON format representation. We choose this format because
(a) JSON is a language-independent, human-readable, and widely used data
format for code-related tasks; (b) it is easily parsable to insert/extract
information; and (c) many web services, APIs, and tools accept JSON objects
and generate responses in JSON format.

In Granite-20B-FunctionCalling, we unify the model output representation of
function calls to the following format:

⬇

{

"name": "<FUNCTION-NAME>",

"arguments": {"<PARAMETER-1>": "VALUE-1", "<PARAMETER-2>": ...}

}

With a similar JSON representation, we express the functions in the library to
be passed as input to the model:

⬇

{

"name": "<FUNCTION-NAME>",

"description: "<FUNCTION-DESCRIPTION>"

"arguments": {"<PARAMETER-1>": {"description": "...", ...} ... }

}

###  3.2 High-Level Function Calling Tasks

In general, these tasks are challenging for LLMs to accomplish since they
require the LLM to generate multiple function calls with the parameters and
their values. In the following sections, we describe different types of high-
level function-calling tasks with examples.

##### Nested Function Calling

The main characteristic of this task is in the output function sequence, where
the current function’s output becomes an input to the next function. So, the
answer to a user query is a sequence of nested function calls selected from
the function library. Furthermore, the parameters of these function calls need
to be filled by extracting the values from the user query. The example below
lists two functions in the output under <function_call> tag 101010In all of
our examples, we use <function_call> tag to represent a function in JSON form.
and the output of the first function will populate the destination parameter
in the second function.

⬇

{

"input": "What’s the typical driving time between Las Vegas and the Grand
Canyon?",

"output: "<function_call> {

"name": "get_location",

"arguments":

{"point_on_map": "the Grand Canyon"}

}

<function_call> {

"name": "get_estimated_duration",

"arguments":

{"source": "Las Vegas",

"method_travel": "driving",

"destination": "<function_response>get_location"}

}"

}

##### Function Chaining

In this task, a model needs to call multiple functions in a sequence to answer
a user query. However, unlike Nested Function Calling, these functions do not
have to be nested. Also, for each function, the parameters whose values are
present in the user query must be passed as arguments. The following example
demonstrates how three functions must be called in sequence to answer the
query.

⬇

{

"input": "Find a provider in SF. Get the weather in SF on March 3rd. Book an
appointment with A businessman’s haircut on March 3rd at 18:15.",

"output: "<function_call> {

"name": "find_provider",

"arguments":

{"city": "SF"}

}

<function_call> {

"name": "get_weather",

"arguments":

{"city": "SF",

"date": "march 3rd"}

}

<function_call> {

"name": "book_appointment",

"arguments":

{"appointment_date": "march 3rd",

"appointment_time": "18:15",

"stylist_name": "A businessman’s haircut"}

}"

}

##### Parallel Functions

Similar to the Function Chaining task, here, the answer to a user query
requires the same function to be called multiple times (in parallel). Also,
the parameters should be mapped with the values extracted from the user query.
In the below example, the get_weather function has been called twice with
different parameter-value pairs to answer the user query.

⬇

{

"input": "What is the weather like in the next 3 days in New York and in the
next 5 days in Los Angeles?",

"output: "<function_call> {

"name": "get_weather",

"arguments":

{"current_location": "New York",

"time_range": "next 3 days"}

}

<function_call> {

"name": "get_weather",

"arguments":

{"current_location": "Los Angeles",

"time_range": "next 5 days"}

}"

}

###  3.3 Low-Level Function Calling Tasks

Compared to High-Level tasks, these tasks are simpler because they either
request function names (without parameters) or parameter-value pairs for a
function. To excel in High-Level function calling tasks, it is crucial for any
LLM to master these foundational sub-tasks. In what follows, we demonstrate
three types of low-level function-calling tasks.

##### Next-Best Function

In this task, given the function library along with the user query and the
partial function sequence, the models are supposed to select the next most
suitable function from the function library. It only requires the model to
choose one function name without any parameters. The following example
demonstrates how the partial function sequence (provided under
<|partial_function_call_list|>) can be passed with the query and the model is
supposed to predict the next best function.

⬇

{

"input": "Get the dates for Raiders Vs Lions in Oakland. Buy 4 tickets for
Raiders Vs Lions in Oakland tomorrow.

<|partial_function_call_list|>

<function_call> {"name": "get_event_dates", "arguments": {}}",

"output: "<function_call> {"name": "buy_event_tickets", "arguments": {}}"

}

##### Function Name Detection

This task expects the model to produce only the sequence of function names
(without parameters) from the function library that are required to answer the
user query. This task closely resembles Function Chaining (a High-Level task),
with the sole distinction being it does not necessitate the model to populate
the function’s arguments. Below is an example of this task.

⬇

{

"input": "Find the hotel named autumn house. Book a hotel named Autumn House
for 2 people for 4 days on Sunday. Find a restaurant in south with name taj
tandoori and price range expensive.",

"output: "<function_call> {"name": "find_hotel", "arguments": {}}

<function_call> {"name": "book_hotel", "arguments": {}}

<function_call> {"name": "find_restaurant", "arguments": {}}"

}

##### Parameter-Value Pair Detection

In this task, when provided with a user query or a user-agent conversation
along with a list of parameters and their descriptions, the model must
identify all the parameters for which the values are present in the query or
conversation. See an example of this task below. To keep the data format
consistent and JSON parsable, we use the function name - “dummy” for all the
data under this task.

⬇

{

"input": "Get a luxury ride to Greentree Apartments with 1 seat.",

"output: "<function_call> {

"name": "dummy",

"arguments":

{"destination": "Greentree Apartments",

"number_of_seats": "1",

"ride_type": "Luxury"}

}"

}

###  3.4 Response Generation

Natural language response generation is a crucial feature of any LLM. In this
task, the model must comprehend an ongoing conversation between a user and an
AI assistant. Then, it generates a natural language response, answering the
most recent user utterance. Such responses are needed to chit-chat with the
user, ask clarifying questions, or synthesize a function call’s output into a
natural language response.

⬇

{

"USER": "I have some chicken, broccoli, and cheese. What can I make with
these?",

"AGENT": "<function_call> {

"name": "search_recipes",

"arguments":

{"ingredients": ["chicken", "broccoli", "cheese"]}

}

<function_response> {

"recipes": [

{"name": "Chicken and Broccoli Casserole",

"ingredients": ["chicken", "broccoli", "cheese"],

"instructions": "Preheat oven to 350 degrees F (175 degrees C). In a large
bowl, mix together chicken, broccoli, and cheese. Bake in preheated oven for
25 to 30 minutes, or until cheese is melted and bubbly."},

{"name": "Cheesy Chicken and Broccoli Pasta",

"ingredients": ["chicken", "broccoli", "cheese"],

"instructions": "Cook pasta according to package instructions. In a large
skillet, cook chicken until no longer pink. Add broccoli and cheese, cook
until cheese is melted."}]

}"

"ASSISTANT": "Here are a couple of recipes you can make with chicken,
broccoli, and cheese:..."

}

##  4 Instruct Tuning

###  4.1 Training Data Mixture Creation

After generating the data for various tasks, the next step is to create a
training data mixture including all the data. We programmatically generate the
mixture of data by following a weighted configuration for datasets and tasks.
Following is an example of the weighted configuration, where the total mixture
samples will be divided between Function Chaining and Next-Best Function in a
3:5 ratio. Within the Function Chaining portion, the allocation is split
between SeqSGD and Glaive-V2 in a 2:3 ratio. Similarly, the Next-Best Function
chunk will be divided in a 2:1 ratio between SeqTopV2 and SeqSNIPS.

⬇

[{

"instruction_name": "Function Chaining",

"datasets": {

"SeqSGD": 2,

"Glaive-V2": 3

},

"weight": 3

},

{

"instruction_name": "Next-Best Function",

"datasets": {

"SeqTopV2": 2,

"SeqSNIPS": 1

},

"weight": 5

}]

Task |  Instruction
---|---
Nested Function Calling |
Function Chaining |
Parallel Functions |  ⬇ SYSTEM: You are a helpful assistant with access to the following function calls. Your task is to produce a sequence of function calls necessary to generate response to the user utterance. Use the following function calls as required.\n<|function_call_library|>\n{API_SPEC_INSTRUCTION}\n\nUSER: {QUERY}\nASSISTANT:
Next-Best Function |  ⬇ SYSTEM: You are a helpful assistant with access to the following function calls. Your task is to produce the next function call necessary to generate response to the user utterance given the partial function list. Use the following function calls as required and return only function "name" with empty "arguments" dictionary in your response. Once all the necessary functions are called, please return "<|endoftext|>".\n<|function_call_library|>\n{API_SPEC_INSTRUCTION}\n\nUSER: {QUERY}\nASSISTANT:
Function Name Detection |  ⬇ SYSTEM: You are a helpful assistant with access to the following function calls. Your task is to produce a sequence of function calls necessary to generate response to the user utterance. Use the following function calls as required and return only function "name" with empty "arguments" dictionary in your response. If no function is relevant, please return "<no_function_call>" followed by "<|endoftext|>".\n<|function_call_library|>\n{API_SPEC_INSTRUCTION}\n\nUSER: {QUERY}\nASSISTANT:
Parameter-Value Pair Detection |  ⬇ SYSTEM: You are a helpful assistant with access to the following function calls. Your task is to find all the necessary arguments and their values from the user utterance to generate response. Use the following function calls as required and fill only the arguments whose values are present in the user utterance.\n<|function_call_library|>\n{API_SPEC_INSTRUCTION}\n\nUSER: {QUERY}\nASSISTANT:
Response Generation |  ⬇ SYSTEM: You are a helpful assistant with access to the following function calls. Your task is to understand the given conversation with function calls and responses and generate natural language response as the ASSISTANT to continue the conversation. You may use the following function calls to understand how to respond to the user query.\n<|function_call_library|>\n{API_SPEC_INSTRUCTION}\n\n{CONV}\nASSISTANT:

Table 2: Task specific instructions

Also, in this step, the training data is embedded with the instructions. These
instructions are based on the tasks associated with the data. Table 4.1
showcases all the instructions (task-wise) we have used in our training. The
“`<|function_call_library|>`” tag has been used for the function library
demonstrated in the prompt with the placeholder named -
`{API_SPEC_INSTRUCTION}`. As the name suggests, the `{QUERY}` and
`{CONVERSATION}` serve as placeholders for user queries or a user-agent
conversation, respectively.

###  4.2 Training

Granite-20B-FunctionCalling is instruct-tuned version of Granite-20B-Code-
Instruct (Mishra et al., 2024)111111https://huggingface.co/ibm-
granite/granite-20b-code-instruct. For training data, we created a mixture of
142K examples spanning all the tasks’ datasets discussed above. We then
trained our model using QLoRA fine-tuning (Dettmers et al., 2023) based on our
multi-task training mixture discussed above. In particular, we trained
Granite-20B-FunctionCalling a QLoRA rank of 8, alpha of 32 and a dropout of
0.1. We also used a learning rate of 5e-5 and ApexFusedAdam as our optimizer
with a linear learning rate scheduler. Training was done using a single node
of 8 A100_80GB GPUs with 800GB of RAM for a total of 3 epochs.

##  5 Experimental Setup and Evaluation

In the section below, we detail our extensive evaluation on various evaluation
datasets and public leaderboard. We provide a comprehensive comparison of our
Granite-20B-FunctionCalling, open sourced with Apache 2.0 license, to other
open and proprietary function calling models.

###  5.1 Datasets

The evaluation datasets and leaderboards for function calling are gaining a
lot of traction in the recent past. In particular, to evaluate the models’
generalizability, we evaluated Granite-20B-FunctionCalling on a variety of
function calling benchmarks, all of which are out-of-domain evaluation for our
model. It is worth noting that some of these datasets; e.g. ToolAlpaca and
ToolLLM, have training data releases. However, we did not use any of these
benchmarks to train Granite-20B-FunctionCalling and we only used the datasets
in 1. 121212We could not verify whether some (or all) of the out-of-domain
datasets were used in other models’ training sets. Table 3 depicts the details
of the evaluation datasets we used. We list the details of each of these
evaluation datasets below.

Dataset | Test Instances | Testing tasks | Metrics
---|---|---|---
BFCL | 1,700 | Function Calling | AST, Execution Accuracy
|  | Relevancy | Accuracy
ToolLLM | 491 | Function Calling | Func. matching (F1)
RestGPT | 157 | Function Calling | Func. matching (F1)
API-Bank | 473 | Function Calling | Func. and Param. matching (F1)
| 478 | Response Generation | BERTscore, ROUGE, BLEU
ToolBench | 214 | Function Calling | Func. and Param. matching (F1)
ToolAlpaca | 100 | Function Calling | Func. and Param. matching (F1)
NexusRaven | 318 | Function Calling | Func. and Param. matching (F1)

Table 3: Evaluation Datasets

(1) Berkeley Function-Calling Leaderboard (BFCL)
131313BFCL:https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html
is a comprehensive function calling leaderboard that includes a dataset of
over 1,700 instances. The leaderboard evaluates tasks that include (a) Simple
Function, Multiple Function, Parallel Function, and Parallel Multiple Function
for Python Language; and (b) for non-Python, they evaluate function relevance
detection, REST API, JavaScript, and Java. (2) ToolBench (Xu et al., 2023b) is
a subset of the data in ToolBench (as released by the authors) focused on
HomeSearch and Booking domains. (3) ToolLLM (Qin et al., 2023)141414ToolLLM
also calls their benchmark ToolBench. To disambiguate, in this paper we use
the term ToolLLM to refer to their benchmark dataset. is synthetically
generated using ChatGPT. The approach uses an initial collection of 16,000
APIs from RapidAPI151515https://rapidapi.com/ for synthetic data generation.
The evaluation is done on the three test sets categorized based on complexity;
G1 – single-tool, G2 – intra-category multi-tool, G3 – intra-collection multi-
tool. (4) RestGPT Song et al. (2023) is a function calling dataset that has
157 test examples with 85 APIs from Spotify and TMDB. This dataset focuses
only on testing model’s ability to detect function names. (5) API-Bank (Li et
al., 2023b) has 314 tool-use dialogues with 753 API calls to assess LLMs’
capabilities in planning, retrieving, and calling APIs. (6) ToolAlpaca (Tang
et al., 2023) is a synthetic data generation approach that has both training
and evaluation benchmarks. It contains 271 tool-use instances spanning 50
distinct categories. Similar to Nexusflow.ai (2023), we used the simulated
part of ToolAlpaca which has a total of 100 test examples. (7) NexusRaven API
Evaluation161616https://huggingface.co/datasets/Nexusflow/NexusRaven_API_evaluation
is another function calling dataset with 318 test examples covering a total of
65 different APIs.

###  5.2 Evaluation Metrics

Below, we define the metrics we adopted for specific tasks in function
calling.

BFCL
Metrics171717https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html#metrics:
BFCL evaluates multiple tasks using the following four metrics. (1) AST
summary compares the abstract syntax tree of the function output to the ground
truth and the function definition. It captures the correctness of the
functions called, their parameters (required or not), and the parameter types.
(2) Execution Summary compares the execution output from generated and ground-
truth function calls. This metric is used to evaluate REST APIs and non-REST
data samples. (3) Relevance evaluates the model’s ability to detect no
function calls when the given list of functions is irrelevant to the user
query. This inversely captures the hallucination rate of models. (4) Overall
Accuracy is the weighted average of all individual data splits in BFCL.

The same metrics described above cannot be used for our out-of-domain datasets
because of missing information, varied formats, and response generation task.
For example, ToolLLM datasets has missing arguments, ToolAlpaca has missing
argument types, and API-Bank has response generation task. Therefore, we use
the following metrics to evaluate the models on other datasets:

F1 measure: Based on Basu et al. (2024), we opted for standard metrics like
precision, recall, and F1 scores which focus on exactly matching API and
parameters’ names. The reason behind this is that APIs are very specific and
unless everything (e.g., name, parameters, input/output format, etc.) matches
the API specifications, executing such APIs will not be possible. We report F1
for matching function names as well as parameter names and values.

Longest Common Subsequence (LCS) and Exact match: We also used LCS from (Basu
et al., 2024) to capture the overlap between the gold and predicted sequences
of APIs. This allows us to compute models’ ability to predict APIs in the
correct sequence as required by the user. Similarly, exact match score (Basu
et al., 2024) checks if all APIs are predicted by the model and are in the
same order.

BERTScore, ROUGE-L and BLEU: We follow the evaluation in API-Bank (Li et al.,
2023b), a dialog dataset that also evaluates model responses based on language
generation metrics such as Rouge-L (Lin, 2004), BertScore (Zhang et al.,
2019), and BLEU (Papineni et al., 2002).

Hallucination Rate: We compute the hallucination rate as the number of samples
where the model predicted an API not provided in the function library.

###  5.3 Evaluation Results

Tables 4, 5, 6, 7, and Figure 3 depicts an extensive evaluation of
Granite-20B-FunctionCalling model in comparison to other state of the art
function calling models. In order to detail this evaluation and analyses,
below we categorize the results into (a) Berkeley Function Calling Leaderboard
Evaluation, and (b) Function calling academic benchmarks.

Model | Organization | License | AST Summary | Exec. Summary | Relevance | Overall Acc.
---|---|---|---|---|---|---
Claude-3.5-Sonnet-20240620 (Prompt) | Anthropic | Proprietary | 91.31 | 89.50 | 85.42 | 90.00
GPT-4-0125-Preview (Prompt) | OpenAI | Proprietary | 91.22 | 88.10 | 70.42 | 88.00
Gemini-1.5-Pro-Preview-0514 (FC) | Google | Proprietary | 87.92 | 83.32 | 89.58 | 86.35
Granite-20B-FunctionCalling | IBM | Apache 2.0 | 84.11 | 86.50 | 87.08 | 84.71
Gorilla-OpenFunctions-v2 (FC) | Gorilla | Apache 2.0 | 89.38 | 81.55 | 61.25 | 84.71
Meta-Llama-3-70B-Instruct (Prompt) | Meta | MetaLlama 3 | 87.74 | 85.32 | 69.17 | 83.88
FireFunction-v2 | Fireworks | Apache 2.0 | 86.44 | 80.26 | 56.67 | 81.88
Mistral-Medium-2312 (Prompt) | Mistral AI | Proprietary | 83.76 | 73.47 | 88.33 | 81.35
Functionary-Medium-v2.4 (FC) | MeetKai | MIT | 85.61 | 75.71 | 74.17 | 80.47
Command-R-Plus (Prompt) (Opt.) | Cohere | cc-by-nc-4.0 | 83.60 | 86.74 | 54.17 | 80.35
Functionary-Small-v2.4 (FC) | MeetKai | MIT | 83.55 | 76.31 | 67.92 | 79.94
Mistral-large-2402 (FC Auto) | Mistral AI | Proprietary | 64.73 | 60.01 | 84.17 | 68.76
Nexusflow-Raven-v2 (FC) | Nexusflow | Apache 2.0 | 65.19 | 73.89 | 57.5 | 67.35
DBRX-Instruct (Prompt) | Databricks | Databricks | 66.62 | 74.92 | 55.83 | 65.88
Snowflake-arctic-Instruct (Prompt) | Snowflake | Apache 2.0 | 61.09 | 80.04 | 59.58 | 65.18

Table 4: Berkeley Function Calling Benchmark: Top 15 models by Overall
Accuracy (as of 06/25/2024). All evaluations are done in a zero-shot manner.

####  5.3.1 BFCL Leaderboard Evaluation Results

Table 4 shows that Granite-20B-FunctionCalling is ranked fourth on the overall
accuracy metric among the top 15 models on BFCL and is highest among models
with open licenses181818We have picked the best performing version of each
model. For example, Gemini-1.5-Pro-Preview-0514 (FC) and Gemini-1.5-Pro-
Preview-0409 (FC) are both part of the leaderboard but for our evaluation, we
consider the best of Gemini-1.5-Pro.. While it is tied with the Gorilla (Patil
et al., 2023) model, it is important to note that the latter was finetuned on
data that are (a) generated from ChatGPT, and (b) similar data to the test set
and hasn’t generalized well to other datasets as shown in Table 5 and Figure
3. In the context of model sizes, Granite-20B-FunctionCalling is one of the
smallest models in the list. Specifically, the ones better than
Granite-20B-FunctionCalling in the ranking are all significantly larger in
size.

For the BFCL evaluation dataset, we highlight concerns in certain categories,
particularly the Java, JavaScript, and REST API evaluations. We are concerned
with how the Java and JavaScript categories evaluate a function-calling
model’s capabilities to follow language-specific syntax, for instance how
objects are instantiated and called in Java and JavaScript utilizing language-
specific context and norms. For the REST API category, we observed significant
brittleness in the evaluation due to issues with API availability and API call
limits.

####  5.3.2 Function Calling Academic Benchmarks

Tables 5 and 6 focus on evaluating the models’ performance on Function
Matching using F1-measure, LCS, and Exact Match. In this experiment, we reuse
the model handlers from the BFCL code base, including the optimized prompts
for each model. However, since the Cohere Command-R-v01 and Mistral-
Instruct-v0.3 handlers available in BFCL use the REST API interface for
inference, we reimplement handlers for these models, utilizing local models
using prompts suggested by the respective model developers for function
calling.

Function Name Detection: On ToolLLM datasets (G1, G2, and G3) and RestGPT,
Granite-20B-FunctionCalling performs the best on detecting function names
given a natural language utterance with 8% better F1 score than the next best
function calling model, as shown in Table 5. Since these datasets have
multiple functions in sequence, we also compute sequencing metrics; exact
score and LCS. On this front, Granite-20B-FunctionCalling model also
outperforms other function calling models by 7% on LCS and 11% on Exact Match
scores.

| ToolLLM-G1 | ToolLLM-G2 | ToolLLM-G3 | RestGPT | Average
---|---|---|---|---|---
| Func.
---
Match
LCS |  | Exact
---
Score
| Func.
---
Match
LCS |  | Exact
---
Score
| Func.
---
Match
LCS |  | Exact
---
Score
| Func.
---
Match
LCS |  | Exact
---
Score
| Func.
---
Match
LCS |  | Exact
---
Score
Functionary-small-v2.4 (7B) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.29 | 0.30 | 0.06 | 0.07 | 0.07 | 0.02
Gorilla-openfunctions-v2 (7B) | 0.59 | 0.59 | 0.28 | 0.48 | 0.48 | 0.22 | 0.51 | 0.52 | 0.24 | 0.21 | 0.21 | 0.01 | 0.44 | 0.45 | 0.19
Hermes-2-Pro-Mistral (7B) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.03 | 0.03 | 0.01 | 0.01 | 0.01 | 0.00
Mistral-Instruct-v0.3 (7B) | 0.49 | 0.49 | 0.26 | 0.51 | 0.49 | 0.30 | 0.36 | 0.33 | 0.13 | 0.36 | 0.37 | 0.08 | 0.43 | 0.42 | 0.19
CodeGemma-Instruct (7B) | 0.59 | 0.59 | 0.21 | 0.53 | 0.53 | 0.13 | 0.52 | 0.54 | 0.16 | 0.22 | 0.23 | 0.02 | 0.46 | 0.47 | 0.13
Nexusflow-Raven-v2 (13B) | 0.65 | 0.65 | 0.39 | 0.73 | 0.72 | 0.43 | 0.68 | 0.66 | 0.27 | 0.39 | 0.41 | 0.06 | 0.61 | 0.61 | 0.28
C4AI-Command-R-v01 (35B) | 0.65 | 0.64 | 0.39 | 0.73 | 0.71 | 0.45 | 0.69 | 0.68 | 0.23 | 0.59 | 0.60 | 0.22 | 0.66 | 0.66 | 0.32
Meta-Llama-3-70B-Instruct (70B) | 0.61 | 0.61 | 0.31 | 0.59 | 0.58 | 0.21 | 0.65 | 0.64 | 0.23 | 0.22 | 0.22 | 0.01 | 0.52 | 0.51 | 0.19
Granite-20B-FunctionCalling | 0.86 | 0.85 | 0.63 | 0.84 | 0.82 | 0.58 | 0.76 | 0.73 | 0.35 | 0.51 | 0.52 | 0.15 | 0.74 | 0.73 | 0.43

Table 5: Function Calling Academic Benchmarks: Function Name Detection. Best
performance is highlighted in bold, second best is underlined. All evaluations
are done in a zero-shot manner.

Full Function Calling: Table 6 reports on the models’ performance on the API-
Bank, ToolBench, and ToolAlpaca datasets that are out-of-domain and evaluated
in a zero-shot manner. No single model outperforms all other models across
datasets. Note that datasets like ToolAlpaca and API-Bank come with training
data split which we never used for training Granite-20B-FunctionCalling, but
could not guarantee that the other models were not trained with it too.
Averaging out the F1 scores across datasets shows that
Granite-20B-FunctionCalling achieves an F1 score of 0.87 when predicting the
function name; second best by 0.01 to Cohere’s Command-R (a 35B model) which
provides an F1 score of 0.88. When predicting the arguments,
Granite-20B-FunctionCalling average F1 score lags behind the best model
(Cohere’s Command-R) by 0.03; 0.62 vs. 0.59.

Function Name Hallucination: Hallucinations have been a major drawback of
large language models. In the context of calling and executing APIs,
hallucinations can have adverse consequences. In Figure 3, we compare the
models’ Function Name Detection Scores (average F1) over all the datasets
(except BFCL, which uses AST-based metrics) and their hallucination rates.
Ideally, we want models to have high performance and low hallucination rates
placing them in the top left corner of the plot. Granite-20B-FunctionCalling
has the highest performance with less than 0.1 hallucination rate.

| Func-Name+Args Det. (F1 Func-Name | F1 Args) | F1 Average
---|---|---
| API-Bank
---
L-1
| API-Bank
---
L-2
| ToolBench
---
HS
| ToolBench
---
B
| Tool-Alpaca
---
| Nexus
---
Raven
| Func
---
Name
Args
Functionary-small-v2.4 (7B) | 0.78 | 0.70 | 0.54 | 0.45 | 0.73 | 0.68 | 0.65 | 0.33 | 0.88 | 0.47 | 0.82 | 0.64 | 0.73 | 0.55
Gorilla-openfunctions-v2 (7B) | 0.43 | 0.41 | 0.12 | 0.12 | 0.86 | 0.69 | 0.41 | 0.27 | 0.69 | 0.39 | 0.81 | 0.65 | 0.55 | 0.42
Hermes-2-Pro-Mistral (7B) |  0.93 | 0.77 | 0.54 | 0.25 | 0.51 | 0.40 | 0.56 | 0.26 | 0.80 | 0.26 | 0.90 | 0.63 | 0.71 | 0.43
Mistral-Instruct-v0.3 (7B) | 0.79 | 0.69 | 0.69 | 0.46 | 0.60 | 0.47 | 0.04 | 0.16 | 0.33 | 0.33 | 0.71 | 0.54 | 0.53 | 0.44
CodeGemma-Instruct (7B) | 0.77 | 0.57 | 0.59 | 0.38 | 0.65 | 0.50 | 0.54 | 0.22 | 0.59 | 0.31 | 0.84 | 0.68 | 0.66 | 0.44
Nexusflow-Raven-v2 (13B) | 0.51 | 0.42 | 0.28 | 0.22 |  0.92 | 0.65 | 0.89 | 0.35 | 0.85 | 0.37 | 0.92 | 0.75 | 0.73 | 0.46
C4AI-Command-R-v01 (35B) |  0.93 | 0.76 | 0.77 | 0.54 | 0.85 | 0.77 | 0.88 | 0.49 |  0.90 | 0.42 |  0.93 | 0.71 | 0.88 | 0.62
Meta-Llama-3-70B-Instruct (70B) | 0.85 | 0.67 | 0.69 | 0.52 | 0.91 | 0.86 |  0.91 | 0.56 | 0.78 | 0.43 | 0.70 | 0.52 | 0.81 | 0.59
Granite-20B-FunctionCalling | 0.91 | 0.71 |  0.83 | 0.60 | 0.87 | 0.71 | 0.82 | 0.36 | 0.89 | 0.44 | 0.92 | 0.72 | 0.87 | 0.59

Table 6: Function Calling Academic Benchmarks: Full Function Calling. Best
performance is highlighted in bold, second best is underlined. All evaluations
are done in a zero-shot manner.

| API-Bank-Response-Level 1 | API-Bank-Response-Level 2
---|---|---
Models | BertScore | Rouge-L | BLEU | BertScore | Rouge-L | BLEU
Functionary-small-v2.4 (7B) | 0.34 | 0.23 | 0.05 | 0.35 | 0.23 | 0.05
Gorilla-openfunctions-v2 (7B) | 0.56 | 0.33 | 0.32 | 0.51 | 0.26 | 0.25
Hermes-2-Pro-Mistral (7B) | 0.45 | 0.18 | 0.09 | 0.42 | 0.14 | 0.06
Mistral-Instruct-v0.3 (7B) | 0.52 | 0.29 | 0.22 | 0.46 | 0.20 | 0.14
CodeGemma-Instruct (7B) | 0.14 | 0.03 | 0.00 | 0.09 | 0.02 | 0.01
Nexusflow-Raven-v2 (13B) | 0.41 | 0.16 | 0.11 | 0.38 | 0.11 | 0.06
C4AI-Command-R-v01 (35B) | 0.39 | 0.15 | 0.07 | 0.39 | 0.15 | 0.06
Meta-Llama-3-70B-Instruct (70B) | 0.69 | 0.48 | 0.47 | 0.65 | 0.40 | 0.40
Granite-20B-FunctionCalling | 0.68 | 0.47 | 0.47 | 0.61 | 0.36 | 0.37

Table 7: API-Bank Response generation dataset evaluation. Results are averaged
across each dataset per model. Best performance is highlighted in bold, second
best is underlined. All evaluations are done in a zero-shot manner. Figure 3:
Performance vs. Hallucination rates for Out-of-Domain Function Calling

####  5.3.3 Response Generation

In Table 7, we show models’ performance on response generation task. We use
API-Bank dataset and follow their response generation task evaluation with
BertScore, Rouge-L, and BLUE. Meta-Llama-3-70B-Instruct has the best
performance across the three metrics with Granite-20B-FunctionCalling coming
in close second (difference in performance ranged between 1-5%). Both models
significantly outperform all other evaluated models. The gap is larger when we
compare Granite-20B-FunctionCalling to the ones specifically trained for
function calling such as Functionary-small-v2.5 and Gorilla-openfunctions-v2.

####  5.3.4 Further Improvements

We have instruct-tuned the Granite-20B-FunctionCalling in such a way that it
develops implicit function searching capability from a long list of functions.
For example, in out-of-domain evaluation tasks, for ToolAlpaca the model needs
to find the Function from a list of 94 functions, similarly, it has access to
15 and 20 functions for ToolBench-B and ToolBench-HS, respectively. Due to
this reason, the prompt with all the function libraries increases the context
length and Granite-20B-Code-Instruct supports up to 8192 context length, so we
were not able to add the full signature of each Function in the library.
Currently, each function in the library contains a function name, a
description, and for each function, we have provided a list of respected
arguments with their descriptions. However, to fit the prompt in the max
context-length, we had to remove the type, required-fields, and optional-
fields values for each argument from the specifications. For further
performance exploration, we will assess options to include the entire function
specification without truncation including exploiting the benefit of Rotary
Position Embedding (Su et al., 2023) and its innate support for longer context
lengths within some of the other Granite models.

##  6 Conclusion

In this paper, we introduced Granite-20B-FunctionCalling, a capable function
calling open model with Apache 2 license. Granite-20B-FunctionCalling is
trained using a suite of datasets transformed from semantic parsing, task-
oriented dialog, personal assistants and conversational domains. The training
setup is a multi-task learning approach where granular tasks in function
calling such as function detection, parameter detection, sequencing, and next
best function are used for instruction tuning the model. We performed an
extensive evaluation of Granite-20B-FunctionCalling in comparison to other
state-of-the-art function calling models. On multiple out-of-domain datasets,
including Berkeley Function Calling Leaderboard, Granite-20B-FunctionCalling
provides the best performance among the models that have open licenses. Even
compared to multiple proprietary models with much larger sizes,
Granite-20B-FunctionCalling showed on-par and in some cases better performance
on multiple datasets and tasks.

## References

  * AI@Meta (2024) AI@Meta.  Llama 3 model card.  2024\.  URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.
  * Basu et al. (2024) Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, and Luis A. Lastras.  Api-blend: A comprehensive corpora for training and benchmarking api llms, 2024.
  * Chen et al. (2024a) Sijia Chen, Yibo Wang, Yi-Feng Wu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Lijun Zhang.  Advancing tool-augmented large language models: Integrating insights from errors in inference trees, 2024a.
  * Chen et al. (2024b) Wei Chen, Zhiyuan Li, and Mingyuan Ma.  Octopus: On-device language model for function calling of software apis, 2024b.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * CodeGemma Team et al. (2024) CodeGemma Team, Ale Jakse Hartman, Andrea Hu, Christopher A. Choquette-Choo, Heri Zhao, Jane Fine, and Hui.  Codegemma: Open code models based on gemma.  2024\.  URL https://goo.gle/codegemma.
  * CohereForAI (2024) CohereForAI.  C4ai command-r: A 35 billion parameter generative model for reasoning, summarization, and question answering.  _Hugging Face Models_ , 2024.  URL https://huggingface.co/CohereForAI/c4ai-command-r-v01.
  * Crouse et al. (2023) Maxwell Crouse, Ibrahim Abdelaziz, Kinjal Basu, Soham Dan, Sadhana Kumaravel, Achille Fokoue, Pavan Kapanipathi, and Luis Lastras.  Formally specifying the high-level behavior of llm-based agents.  _arXiv preprint arXiv:2310.08535_ , 2023.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  Qlora: Efficient finetuning of quantized llms, 2023.
  * Gao et al. (2023) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.  Pal: Program-aided language models.  In _International Conference on Machine Learning_ , pp. 10764–10799. PMLR, 2023.
  * Guo et al. (2024) Zhen Guo, Adriana Meza Soria, Wei Sun, Yikang Shen, and Rameswar Panda.  Api pack: A massive multi-programming language dataset for api call generation, 2024.
  * Gur et al. (2023) Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust.  A real-world webagent with planning, long context understanding, and program synthesis.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * He-Yueya et al. (2023) Joy He-Yueya, Gabriel Poesia, Rose E Wang, and Noah D Goodman.  Solving math word problems by combining language models with symbolic solvers.  _arXiv preprint arXiv:2304.09102_ , 2023.
  * Jain et al. (2024) Arushi Jain, Shubham Paliwal, Monika Sharma, Lovekesh Vig, and Gautam Shroff.  Smartflow: Robotic process automation using llms, 2024.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_ , 2023.
  * Komeili et al. (2022) Mojtaba Komeili, Kurt Shuster, and Jason Weston.  Internet-augmented dialogue generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 8460–8478, 2022.
  * Li et al. (2023a) Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter.  Chain of code: Reasoning with a language model-augmented code emulator.  _arXiv preprint arXiv:2312.04474_ , 2023a.
  * Li et al. (2024) Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al.  Synthetic data (almost) from scratch: Generalized instruction tuning for language models.  _arXiv preprint arXiv:2402.13064_ , 2024.
  * Li et al. (2023b) Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li.  Api-bank: A comprehensive benchmark for tool-augmented llms, 2023b.
  * Lin (2004) Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pp. 74–81, 2004.
  * MeetKai (2024) MeetKai.  Functionary-7b-v1.4: A language model for function interpretation and execution.  _Hugging Face Models_ , 2024.  URL https://huggingface.co/meetkai/functionary-7b-v1.4.
  * Mishra et al. (2024) Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, et al.  Granite code models: A family of open foundation models for code intelligence.  _arXiv preprint arXiv:2405.04324_ , 2024.
  * Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi.  Cross-task generalization via natural language crowdsourcing instructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 3470–3487, 2022.
  * Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.  Crosslingual generalization through multitask finetuning.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 15991–16111, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.891.  URL https://aclanthology.org/2023.acl-long.891.
  * Nexusflow.ai (2023) Nexusflow.ai.  Nexusraven-v2: Surpassing gpt-4 for zero-shot function calling, 2023.  URL https://nexusflow.ai/blogs/ravenv2.
  * Nous-Research (2023) Nous-Research.  Nous-hermes-13b.  https://huggingface.co/NousResearch/Nous-Hermes-13b, 2023.  Model on Hugging Face.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pp. 311–318, 2002.
  * Parisi et al. (2022) Aaron Parisi, Yao Zhao, and Noah Fiedel.  Talm: Tool augmented language models.  _arXiv preprint arXiv:2205.12255_ , 2022.
  * Patel et al. (2021) Arkil Patel, Satwik Bhattamishra, and Navin Goyal.  Are NLP models really able to solve simple math word problems?  In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 2080–2094, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.168.  URL https://aclanthology.org/2021.naacl-main.168.
  * Patil et al. (2023) Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez.  Gorilla: Large language model connected with massive apis.  _arXiv preprint arXiv:2305.15334_ , 2023.
  * Qin et al. (2023) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al.  Toolllm: Facilitating large language models to master 16000+ real-world apis.  _arXiv preprint arXiv:2307.16789_ , 2023.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model, 2023.
  * Reid et al. (2024) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  _arXiv preprint arXiv:2403.05530_ , 2024.
  * Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al.  Code llama: Open foundation models for code.  _arXiv preprint arXiv:2308.12950_ , 2023.
  * Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.  Toolformer: Language models can teach themselves to use tools.  _ArXiv_ , abs/2302.04761, 2023.  URL https://api.semanticscholar.org/CorpusID:256697342.
  * Shinn et al. (2023) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao.  Reflexion: Language agents with verbal reinforcement learning.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Song et al. (2023) Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, Ye Tian, and Sujian Li.  Restgpt: Connecting large language models with real-world restful apis, 2023.
  * Su et al. (2023) Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.  Roformer: Enhanced transformer with rotary position embedding.  2023\.
  * Sudalairaj et al. (2024) Shivchander Sudalairaj, Abhishek Bhandwaldar, Aldo Pareja, Kai Xu, David D Cox, and Akash Srivastava.  Lab: Large-scale alignment for chatbots.  _arXiv preprint arXiv:2403.01081_ , 2024.
  * Tang et al. (2023) Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun.  Toolalpaca: Generalized tool learning for language models with 3000 simulated cases.  _arXiv preprint arXiv:2306.05301_ , 2023.
  * Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-instruct: Aligning language models with self-generated instructions.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 13484–13508, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.754.  URL https://aclanthology.org/2023.acl-long.754.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_ , 2021.
  * Xu et al. (2023a) Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, and Dongkuan Xu.  Rewoo: Decoupling reasoning from observations for efficient augmented language models.  _arXiv preprint arXiv:2305.18323_ , 2023a.
  * Xu et al. (2023b) Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang.  On the tool manipulation capability of open-source large language models.  _arXiv preprint arXiv:2305.16504_ , 2023b.
  * Yan et al. (2024) Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez.  Berkeley function calling leaderboard.  https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html, 2024.
  * Yang et al. (2024a) John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press.  Swe-agent: Agent-computer interfaces enable automated software engineering, 2024a.
  * Yang et al. (2024b) Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan.  Gpt4tools: Teaching large language model to use tools via self-instruction.  _Advances in Neural Information Processing Systems_ , 36, 2024b.
  * Yang et al. (2023) Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.  Mm-react: Prompting chatgpt for multimodal reasoning and action.  _arXiv preprint arXiv:2303.11381_ , 2023.
  * Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao.  React: Synergizing reasoning and acting in language models.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_ , 2019.