xLAM: A Family of Large Action Models to Empower AI Agent Systems
Jianguo Zhang Tian Lan Ming Zhu Zuxin Liu Thai Hoang Shirley Kokane Weiran Yao Juntao Tan Akshara Prabhakar Haolin Chen Zhiwei Liu Yihao Feng Tulika Awalgaonkar Rithesh Murthy Eric Hu Zeyuan Chen Ran Xu Juan Carlos Niebles Shelby Heinecke Huan Wang Silvio Savarese Caiming Xiong


Optimal Regularity for Fully Nonlinear Nonlocal Equations with Unbounded Source Terms
    Disson S. dos Prazeres and Makson S. Santos
    Received 16 July 2024; accepted 04 September 2024
=====================================================================================



§ ABSTRACT 
Autonomous agents powered by large language models (LLMs) have attracted significant research interest. However, the open-source community faces many challenges in developing specialized models for agent tasks, driven by the scarcity of high-quality agent datasets and the absence of standard protocols in this area.
We introduce and publicly release xLAM, a series of large action models designed for AI agent tasks. The xLAM series includes five models with both dense and mixture-of-expert architectures, ranging from 1B to 8x22B parameters, trained using a scalable, flexible pipeline that unifies, augments, and synthesizes diverse datasets to enhance AI agents' generalizability and performance across varied environments.
Our experimental results demonstrate that xLAM consistently delivers exceptional performance across multiple agent ability benchmarks, notably securing the 1st position on the Berkeley Function-Calling Leaderboard, outperforming GPT-4, Claude-3, and many other models in terms of tool use. By releasing the xLAM series, we aim to advance the performance of open-source LLMs for autonomous AI agents, potentially accelerating progress and democratizing access to high-performance models for agent tasks.



Models: https://huggingface.co/collections/Salesforce/xlam-models-65f00e2a0a63bbcd1c2dade4huggingface.co/Salesforce/xLAM-models 

GitHub: https://github.com/SalesforceAIResearch/xLAMgithub.com/SalesforceAIResearch/xLAM






§ INTRODUCTION

The field of autonomous agents has witnessed significant advancements in recent years, with large language models (LLMs) playing a crucial role in enhancing agent capabilities across diverse tasks. Researchers have made substantial progress in developing sophisticated frameworks <cit.> and specialized environments <cit.> to enhance agent capabilities, such as tool use <cit.> and web browsing <cit.>. Concurrently, comprehensive benchmarks like AgentBench <cit.>, ToolBench <cit.>, and AgentBoard <cit.> have been established to rigorously assess agent performance in reasoning, planning, and multi-turn interactions.

While proprietary LLMs developed by industry leaders have demonstrated competitive performance in various agent tasks <cit.>, the open-source community faces limited choices for specialized models in this domain. This scarcity stems from several challenges in adapting open-source LLMs to agent tasks, primarily due to the lack of comprehensive, high-quality datasets and the heterogeneity of existing data formats. These factors complicate the unification of diverse datasets and obstruct the learning of transferable knowledge across different agent tasks.

Recently, the agent research community has intensified efforts in open-source agent data processing and model training <cit.>. 
However, these works still face challenges in managing complex environments and generalizing to new scenarios, primarily due to limitations in the collected agent data.
A major obstacle is the homogeneity of content and format in existing datasets, resulting in models that lack diversity across various tasks and struggle to adapt to new or slightly different data structures in practical applications. While previous efforts have attempted to design pipelines for unifying data, they typically cover only a few scenarios or lack flexibility in their unified formats. For instance, Lumos <cit.> primarily addresses question answering, web agents, and mathematical tasks involving planning and grounding; while AgentOhana <cit.>, despite encompassing a more diverse range of environments, lacks an extendable unified format to accommodate new environments.

Moreover, open-source datasets often suffer from quality issues, such as incorrect agent outputs, hallucinated actions, and repeated interaction turns within trajectories <cit.>. The lack of detailed analysis and understanding of agent data further complicates these challenges, hindering the development of robust and versatile open-source agent models. Addressing these challenges is crucial for advancing the field of open-source agent models and bridging the performance gap with proprietary LLMs in agent tasks.






In this work, we introduce and open-source xLAM, a series of powerful models with varying sizes. This diverse set is tailored for a variety of applications, with smaller models (1B and 7B) optimized for on-device deployment, while larger models (8x7B and 8x22B) are designed to tackle more challenging tasks. Alongside the model release, we offer several insights and lessons learned from our experience in agent model training:

 
    
  * Data Processing: We highlight the importance of data unification and augmentation in enhancing dataset diversity and mitigating overfitting. Our developed dataset preprocess and augmentation pipeline significantly improves the generalizability of agent models across diverse environments.
    
    
  * Data Synthesis: We showcase the impact of scalable, high-quality data synthesis on agent model performance. Our synthetic dataset enabled xLAM models to achieve 4 of the top 20 positions on the Berkeley Function Calling Leaderboard, including securing the top-1 spot (Fig. <ref>), with smaller models achieving performance comparable to much larger counterparts, showing great potential in this direction.




We evaluate the xLAM series on public agent benchmarks, demonstrating exceptional performance across various agent tasks. By open-sourcing these models, we aim to advance open-source agent models and provide valuable insights into data processing and synthesis techniques, addressing key challenges in developing competitive alternatives to proprietary models.




































 
 
 

 
 

 
 

 




§ RELATED WORK





 §.§ LLM Agents


Recent advancements in LLMs have significantly enhanced their utility in various agent tasks. Several innovative prompt techniques have been developed to improve performance, including Chain of Thought (COT) <cit.>, ReACT <cit.>, and Reflection <cit.>. Additionally, considerable efforts have been made to fine-tune open-sourced agent models for better capabilities <cit.>. These include enhancements in data collection and processing to facilitate effective agent learning <cit.>, covering a range from simple question answering to more complex scenarios like web interactions, tool operations, reasoning, and planning. 
However, many of these agent frameworks still depend on proprietary models as their core engine to achieve optimal performance, revealing a substantial gap in the availability of high-quality open-source models for these tasks.



 §.§ Agent Benchmarks


A variety of benchmarks have been established to assess the abilities of LLM agents across diverse scenarios <cit.>. Notably, AgentBench <cit.>, Mint-Bench <cit.>, and AgentBoard <cit.> encompass environments ranging from code generation and games to web interactions and reasoning tasks. ToolBench <cit.> specifically evaluates multi-turn reasoning and tool-usage abilities, while the Berkeley Function-Calling Leaderboard <cit.> broadly assesses models' capabilities in function calling across various contexts. These recent advancements in benchmarking have made the evaluation of agent models more accessible and standardized.






§ DATA PROCESSING PIPELINE




In this section, we discuss the data pipeline for training xLAM, including data unification, augmentation, quality verification, general instruction data synthesis, and preference data generation.











 §.§ Data Unification



Existing agent datasets are collected from diverse environments and designed in various formats, introducing noise and complicating data augmentation and verification. Models like NexusRaven <cit.>, Gorilla-Openfunctions <cit.>, and AgentOhana <cit.> have demonstrated superior performance in function-calling, suggesting that a well-defined, universal format could significantly enhance model performance. By standardizing the format of existing data, we can reduce noise and facilitate easier data augmentation and quality verification, leading to a more efficient and robust framework for model training and evaluation. Furthermore, a standardized format ensures consistency, simplifies model training, and enhances the model's ability to generalize across various benchmarks.

Function-calling formats form the basis for how models understand and execute tasks, motivating us to design our unified data format in a function-calling style. 
As illustrated in Figure <ref>, 
the unified format consists of several modules: task instruction, available tools, format instruction, few-shot examples, query, and steps. 
Specifically, the available tools define the agent's action space, and the format instruction specifies the output format the agent should follow when generating a response. In each step, the agent's output, the environment's feedback/execution results, and the user's follow-up input are organized into a dictionary. 
It's quite common for there to be purely conversational interactions between users and agents that don't trigger any APIs or receive corresponding observations. In these instances, the related entry values would simply remain empty.

This unified format is compatible with various environments and tasks, making our data processing pipeline adaptable to different datasets and scalable to large amounts of data. Moreover, the modularized design allows for fine-grained data augmentation and quality verification, which are essential in improving agent data quality. For example, by unifying all the available tools and tool calls, we can easily inspect for hallucination and function-call errors, and apply various augmentation techniques.





















 §.§ Data Augmentation



Our data augmentation strategy focuses on improving the diversity of the data. It involves applying various transformations to the existing dataset, thereby generating new, synthetic data samples. The data unification step significantly simplifies the application of various augmentation techniques. A standardized data format ensures consistency and ease of implementation, allowing for more efficient augmentation processes. Specifically, the augmentation techniques we adopted can be categorized as prompt format augmentation and instruction-following augmentation.

Prompt Format Augmentation: Prompt format augmentation focuses on creating various prompt formats based on the structured, unified data format. The format augmentation can be further divided into two categories: 1) Order Shuffling. In the unified format, the available tools are provided in a list, and each tool contains the name, description, and parameters. To avoid model overfitting to the specific order of the tools, we randomly shuffle the tool list. Furthermore, we also shuffle the order of the name, description, parameters, and within the parameters to present the information in different ways. We do the same thing within the tool_calls in each step. Additionally, we also shuffle the order of different sections of the input, including task instruction, tools, format instruction, few-shot examples etc. 2) Concatenation Tokens. Each training data point is a pair of input and output sequences. To convert the structured unified format to the training prompt, we use special tokens to concatenate different sections into one sequence. We create several different special token styles, including "[START/END OF QUERY]", "<query></query>", and plain text. 

Instruction-Following Augmentation:
Instruction-following augmentation focuses on adding diversity to the instructions in order to improve the model's instruction-following capability. It involves rephrasing existing instructions and adding new instructions, without introducing inaccuracy and inconsistency. Therefore, verification of the new instructions is a crucial step for this type of augmentation. We employ two methods for instruction-following augmentation: 1) Task Instruction Rephrasing. We rephrase the task instructions using powerful LLMs to accommodate various input styles from users. To ensure the rephrased instructions still align with the original version, we verify them by prompting the LLMs with the rephrased instructions and check if the LLMs can still follow them and generate correct function calls. 2) Format Instruction-Following. In our unified format, the output format is a JSON string with  and . To avoid the model overfitting on JSON format and to enable the model to follow various output formats upon different format instructions, we prepare 15 different output formats along with their corresponding format instructions and format converters. The output formats include JSON, XML, YAML, plain text, etc. 






 §.§ Data Quality Verification


To further understand of the data quality and to thoroughly investigate the sources of errors in the evaluation, we conduct a detailed analysis of the unified dataset. We identify a list of errors in the data using both rule-based and LLM-as-a-judge approaches.


Undefined Function Call: 
In function-calling, a list of available functions is provided, and the model should generate a function_call using one of the given functions. However, we found that in many cases, the predicted function_call is not from the given list. We match the predicted function with the given functions by comparing the function names and the list of parameter names. When the function_call name does not match any given functions, we refer to it as Undefined Functions Invoked. When the function name matches but the argument list contains undefined arguments, we refer to it as Undefined Arguments Passed. We also take into consideration optional parameters. 


Incorrect Argument Type:
Other than the error types mentioned above, we also observe that sometimes the model generates the correct argument's value, but in the wrong types. For example, when a parameter expects a , the generated arguments is , which is a string version of the list. When executing the function call, errors will occur due to incorrect data type.  We identify trajectories containing the incorrect argument type error by comparing the parameter type in the available tools and the actual argument type. We also found that most argument type errors can be fixed by converting the arguments to the correct parameter types. 

Argument Hallucination: Upon examining the unified dataset from public sources, we discovered that tool calls frequently include argument values not present in the user query or prior steps. This issue arises because much of this data is generated by LLMs, which are prone to hallucination, a common problem in LLM-generated content.
We identified two types of hallucination: 1) the generated tool names or argument names do not appear in the provided tool and argument list; and 2) the argument values do not align with the user query or observations from previous steps. The first type of hallucination is straightforward to address by searching the generated tool call and argument names and matching them with the provided tool list, as they are all structured in JSON, making this process efficient. However, detecting the second type, where argument values are misaligned, is more challenging, as simple string matching is ineffective for complex queries and tasks. To tackle this, we use LLMs as judges to perform step-wise argument hallucination detection, detecting if there is a mismatch between the arguments and the intended query or prior observations.

Low-Quality Reasoning and Planning: We observe many data trajectories where the reasoning and planning steps are of low quality, which is a common issue in the outputs of many LLMs. To address this, we first filter out low-quality data using rule-based methods informed by heuristics, then prompt models like Mixtral-8x22b-Instruct-v0.1 <cit.> and DeepSeek-V2 <cit.> to evaluate both the overall trajectory and individual thought steps on the selected data. A portion of these rating results is then sampled and verified by humans. We also attempted to iterate on this process using specifically fine-tuned models.



 §.§ Data Synthesis


Based on our findings in Sec. <ref>, we observe that most of these publicly available datasets have several limitations. First, these datasets are often static, synthesized by weak models, limited in scope, and, more importantly, not verified by execution. 
Second, these datasets mainly focus on a single type of function-calling category, i.e., outputting a single function call based on the provided tools. However, real-world scenarios might consist of many other types of use cases, such as the parallel function-calling scenario <cit.>, where the user query contains multiple requests and the model should respond with concurrent function calls in parallel within a single response.

To address these two issues, we utilize a systematic data synthesis framework called APIGen <cit.>, which can generate verifiable datasets based on a collection of executable APIs.
The key idea is a multi-stage verification process to ensure the accuracy and quality of the generated data. This process includes format verification as introduced in Sec. <ref>, execution verification, and semantic verification, which collectively help to identify and filter out low-quality data points, such as those with hallucination issues or inaccurate argument parameter values.

 
We utilize over 3,673 APIs across 21 categories from ToolBench <cit.> to generate a total of 60,000 high-quality data. These samples are generated using several strong open-source language models: DeepSeek-V2-Chat <cit.> and Mixtral-8x22B-Inst <cit.>. 
This synthesis framework greatly improves the robustness and applicability of the dataset, as the majority of low-quality data can be identified by the multi-stage verification process. 






 §.§ Data Mixture

For supervised fine-tuning (SFT), our dataset combines training samples from three main sources: cleaned and augmented agent datasets, a synthetic function-calling dataset, and general instruction-tuning datasets. These sources are used to train the general xLAM models.  

Specifically, to enhance the general instruction capability of xLAM, we integrate diverse instruction-tuning datasets from DialogStudio <cit.> and Data Provenance <cit.>. We employe rule-based techniques to filter out low-quality data, such as repetitive words and turns, which are common and often produced by less powerful models. We also remove data with inappropriate contents, responses and non-commercial licenses.  Additionally, we deduplicate examples with similar user queries and organized the data by domain or category. We then prompt  Mixtral-8x22b-Instruct-v0.1 and DeepSeek-V2 to assess both the entire dialogue and individual system responses on the selected data. This instruction data comprises 20% to 30% of our training set.  To further enhance model robustness, we preserve the original formats of the general instruction-tuning data. 

To enhance the function-calling capability of xLAM-7b-fc-r and xLAM-1b-fc-r, we employ a targeted training approach, with 50% of their training data drawn from our high-quality synthetic function-calling dataset. The remaining 50% of the training data is sampled from other tasks within our training set.

For Direct Preference Optimization (DPO) <cit.>, we prompt less powerful models to generate and rate responses for selected data from each source, then sample a subset for human verification. After adjustments to models and prompts, we classify the selected responses as rejected samples. 






§ MODEL TRAINING









 §.§ Modeling



We use a supervised fine-tuning 
(SFT) approach, further aligning model checkpoints with the DPO method, and leverage the robustness of our flexible data pipeline. Our training code is based on the HuggingFace Transformers and Accelerate libraries<cit.>, as well as PyTorch FSDP<cit.>. During training, the model undergoes multiple epochs, with datasets randomly shuffled each time. When using data parallelism across multiple devices, we diversify random seeds based on process IDs, ensuring balanced data distribution through partitioning, shuffling, and interleaving, thereby enhancing the robustness and reproducibility of our training process.



The fine-tuning of general xLAM models is conducted on Nvidia H100 GPUs. For SFT, we use a full fine-tuning framework that employs the fully sharded data parallel algorithm <cit.>. In the case of xLAM-8x22b-r, we integrate LoRA <cit.> to better preserve the model's original capacities and prevent catastrophic forgetting <cit.>. LoRA is also used for DPO alignment across all xLAM models. Additionally, we use a cosine learning rate scheduler with 100 warm-up steps to optimize performance.


The xLAM-FC models target various categories of function-calling agents, including simple, multiple, parallel, and parallel multiple. These categories are designed to enhance the models' performance in different scenarios. For instance, a simple query like retrieving the weather for a location (e.g., "What is the weather in Palo Alto today?") can be handled by calling . Multiple queries involve selecting the appropriate function from several APIs, while parallel queries require executing multiple function calls simultaneously. Additionally, the models are trained in relevance detection to ensure alignment between function calls, execution results, and query objectives.






 §.§ xLAM Model Series

We introduce a series of agent models tailored for different use cases. Our flagship model series, xLAM, is built upon the Mixtral Instruct <cit.> models and aims to achieve balanced performance across a diverse range of agent tasks, from complex multi-turn interactions to function-calling applications. To ensure its versatility, xLAM is trained on uniformly sampled data from our training dataset as introduced in Sec. <ref>.

In addition to general xLAM models, we develop two specialized models for function-calling use cases, xLAM-7b-fc-r and xLAM-1b-fc-r, based on DeepSeek-Coder-7B-instruct-v1.5 and DeepSeek-Coder-1.3B-instruct, respectively <cit.>. The smaller model sizes offer increased accessibility, allowing users to easily host them on a single GPU to address various function-calling tasks, ranging from simple user queries to parallel concurrent requests.




By offering a suite of models with varying sizes and specializations, the xLAM series caters to a wide range of user needs and computational resources, making powerful agent capabilities more accessible and adaptable to real-world applications.























§ EXPERIMENTS





 §.§ Benchmarks


After considering the stability of environments and research budget limitations, we evaluate the performance of models across four rigorous benchmarks: Webshop <cit.>, ToolQuery <cit.>, ToolBench <cit.>, and the Berkeley Function-Calling Benchmark <cit.>. Each benchmark is designed to assess different aspects of model capabilities under a variety of settings and constraints.

Webshop is an interactive web environment designed to mimic online shopping experiences, testing an agent's ability to navigate and assist in e-commerce tasks. Webshop comprising approximately 250 test cases. 

ToolQuery evaluates an agent's skills in using tools to retrieve and process information across domains. ToolQuery features 60 test cases across three distinct settings: Weather, Movie, and Academia. 


We use the testing configurations from AgentBoard <cit.> for both Webshop and ToolQuery. These configurations assess overall performance using the Success Rate and evaluate progressive performance across interactive turns with the Progress Rate, with  Success Rate being the more critic metric.

We additionally evaluate on ToolQuery-Unified, which is essentially ToolQuery but requires an agent to ingest the task instruction and tools following the augmented prompt format described in <ref>  and likewise solve the task following the unified format. 
The purpose of testing agents in this setting is to assess their reasoning and tool-use abilities when evaluated on structured formats <cit.>.

ToolBench is developed for real-time evaluation of multi-turn reasoning and interactive capabilities via RapidAPI, and includes around 1,000 test cases. It uses Pass Rate as the metric, where the trajectory and final response are sent to GPT-4-0125-preview to determine whether the agent's final response successfully addresses the given user query. The evaluations cover both in-domain and out-of-domain settings, including unseen instructions with familiar tools, unseen tools within previously known categories, and entirely new categories of unseen tools.


Berkeley Function-Calling Leaderboard (BFCL) Benchmark <cit.> provides a comprehensive evaluation framework for assessing an agent's capability to reason about and execute function calls across a variety of programming languages and application domains. The benchmark comprises over 2,200 test cases, challenging models with complex scenarios such as parallel and multiple function calls in languages like Java, JavaScript, and Python. The evaluation metrics include Abstract Syntax Tree (AST) accuracy for non-executable test queries, executable accuracy by running APIs to obtain results, and a relevance detection score that measures the agent's ability to distinguish non-relevant queries and provided tools.

Importantly, our evaluation utilizes the most recent BFCL v2 version, as of the cutoff date 09/03/2024. The v2 version introduces live function calls and real-world scenarios contributed by users, addressing issues such as data contamination, bias, and fairness by leveraging user-provided data. This updated dataset better reflects real-world distributions, characterized by a higher demand for selecting among multiple functions and a reduced demand for parallel function calls. For instance, our analysis indicates that in the v2 benchmark, the average number of available functions has doubled, while the average number of function calls has been halved compared to the non-live v1 data. It is important to note that all our models were trained prior to the release of the BFCL v2 live data.







 §.§ Experimental Results





  §.§.§ Webshop and ToolQuery



 



Webshop. Table <ref> presents detailed comparisons of state-of-the-art language and agent models in the Webshop and ToolQuery environments, illustrating the robust and strong performance of the xLAM models. In the Webshop environment, xLAM-7b-r not only achieves the highest Success Rate at 0.414, surpassing other general LLMs like GPT-4-0125-preview, GPT-4o-2024-0523, and Claude2, but also outperforms specialized agent models such as AgentOhana-8x7b and Lemur-70b. This demonstrates xLAM models' superior ability to navigate and execute tasks in the web interaction environment effectively.

ToolQuery. In the more complex and unseen ToolQuery environment, xLAM-8x7b-r and xLAM-8x22b-r also demonstrate high performance as shown in Table <ref>, ranking second with a Success Rate of 0.683. This shows a significant improvement over the baseline performance of Mixtral-8x7b-inst and Mixtral-8x22b-inst, which are 0.167 and 0.400, respectively. Notably, all three xLAM models surpass the Mixtral-8x22B-Instruct model. Despite Mixtral-8x22B-Instruct having a large number of parameters and specialized tuning for advanced functionalities such as function calling, reasoning, and complex tool usage, it falls short of the xLAM models’ performance. Furthermore, same as other general LLMs, it lacks transparency regarding the data collection, unification processes, and other critical details, contrasting with the open source purposes provided for xLAM. These results show the efficacy of our proposed data unification and synthetic data pipeline. 



ToolQuery-Unified. When the system prompt from ToolQuery is presented to the model in the unified format shown in Fig. <ref>, and the model is required to follow the provided format instructions to generate a structured output, we observe that xLAM models' performances are more consistent compared to GPT models, as shown in Table <ref>. While GPT-4o’s performance significantly degrades by 42% compared to ToolQuery, our best xLAM 8x22b model maintains comparable performance. This can be attributed to xLAM being trained on trajectories that adhere to the unified format, enabling it to perform consistently during inference. Concurrent research <cit.> observed a similar decline in performance on reasoning tasks when LLMs are constrained to produce output in specific formats. Deeper analysis indicated that the degradation is more than just due to incorrectly formatted output in a specific format, but rather due to a drop in the reasoning ability of the model itself.   









  §.§.§ ToolBench


Table <ref> presents the results on ToolBench, where xLAM models demonstrate impressive performance. They surpass both TooLlama-V2 and GPT-3.5-Turbo-0125 across all test settings. Moreover, xLAM models outperform AgentOhana-8x7b in scenarios involving unseen instructions and unseen tools, while achieving performance comparable to GPT-4-0125-preview in the unseen tools setting. These results show xLAM models' robust capabilities in multi-turn reasoning and complex tool usage, effectively handling both in-domain and out-of-domain tasks.



  §.§.§ Berkeley Function-Calling Benchmark





Table <ref> presents the experimental results on the BFCL v2 benchmark (cutoff date 09/03/2024), which shows the exceptional performance of our xLAM model series in function-calling tasks. Notably, xLAM models secure four out of the top twenty positions, demonstrating the effectiveness of our data pipeline and training methodology across various model sizes.

Our flagship model, xLAM-8x22b-r, achieves the highest overall accuracy of 87.31%, surpassing all other models in the benchmark. This result validates the effectiveness of our data processing and model training pipeline in improving models' function-calling ability. Following closely, xLAM-8x7b-r ranks 6th, outperforming most prominent models including GPT-4o-mini and Claude-3.

The performance of our models demonstrates clear scaling with model size, a trend exemplified by xLAM-7b-r, which ranks 14th with an accuracy of 80.33%. This model outperforms several larger and more resource-intensive alternatives, including multiple GPT-4 and GPT-4o versions, highlighting the potential of small models in the agent area.


Perhaps most remarkably, our smallest model, xLAM-1b-fc-r, achieves a 32nd place ranking with an accuracy of 75.43%, surpassing much larger models like Claude-3-Opus (FC) and GPT-3.5-Turbo. This performance underscores the power of our data synthesis framework in producing high-quality, diverse datasets that enhance function-calling effectiveness even for smaller language models.


It is also worth noting that the BFCL v2 benchmark <cit.> includes a live dataset released after our model training date. These fresh data are collected from real-world user queries that were entirely unseen by our models. Nevertheless, our models exhibit strong generalization capabilities in handling these real-world use cases. 
The consistently strong performance across our model series, ranging from 8x22 billion to 1 billion parameters, demonstrates the scalability and versatility of our approach. This scalability is particularly noteworthy, as it enables strong results from compact models suitable for resource-constrained environments to large-scale models for more demanding applications. Furthermore, the ability of our smaller models to compete with much larger alternatives suggests significant potential for efficient deployment in various real-world scenarios.





 §.§ Ablation Study

r0.55
  


    < g r a p h i c s >

  

    Ablation study for data augmentation and data quality verification (cleaning).
  
  



We conducted an ablation study on the 7B models to measure the impact of various steps in our data pipeline.
Three datasets were prepared for this analysis: raw data, augmented data, and augmented + cleaned data. The raw data represents the dataset before data unification, while the other two datasets are post-unification.
Figure <ref> presents the evaluation results of models trained on these three datasets. The metrics used for this evaluation are G1_instruction from ToolBench and success_rate from both Webshop and ToolQuery. The results indicate that augmented data consistently outperforms raw data across all metrics, with improvements of 2.3% on ToolBench, 5.8% on Webshop, and 18.3% on ToolQuery. Furthermore, the addition of data cleaning leads to a substantial performance increase on ToolQuery, with a further improvement of 23.4%. The results highlight the effectiveness of data augmentation and cleaning processes in the data pipeline.







§ CONCLUSION


This paper introduces xLAM series, a set of large action models for autonomous AI agents. Our models, ranging from 1B to 8x22B parameters, were trained with a scalable and flexible data pipeline that unifies, augments, and synthesizes diverse datasets. 
Our evaluations show that xLAM models consistently perform exceptionally across various benchmarks. 
The insights we learned from training these models highlight the importance of rigorous data processing and the potential of data synthesis in developing capable AI agents.
By releasing the xLAM series to the public, we aim to democratize access to high-performance models for agent tasks, thereby accelerating progress in the field. 



unsrt





§ APPENDIX








    mybox/.style=
        colback=lightgold!50,
        colframe=lightgold!50,
        arc=4mm,
        boxrule=0.5mm,
        left=1mm,
        right=1mm,
        top=1mm,
        bottom=1mm,
        boxsep=1mm,
        coltitle=black,
        sharp corners,
    















