Sketch: A Toolkit for Streamlining LLM Operations
Xin Jiang Xiang Li Wenjia Ma Xuezhi Fang Yiqun Yao Naitong Yu Xuying Meng Peng Han Jing Li Aixin Sun Yequan Wang


Few-Shot Continual Learning for Activity Recognition in Classroom Surveillance Images
 ^† Corresponding author (lfxu@uestc.edu.cn) 
 ^* Equal Contribution 

    
    Yilei Qian^*, Kanglei Geng^*, Kailong Chen, Shaoxu Cheng, Linfeng Xu^†, Hongliang Li, Fanman Meng, Qingbo Wu 

    School of Information and Communication Engineering 

    University of Electronic Science and Technology of China, Chengdu, China 

    {yileiqian, kangleigeng, chenkailong, shaoxu.cheng}@std.uestc.edu.cn, {lfxu, hlli, fmmeng, qbwu}@uestc.edu.cn

    September 9, 2024
===================================================================================================================================================================================================================================================================================================================================================================================




[2]Indicates equal contribution.
[1]Corresponding author.


§ ABSTRACT 
Large language models (LLMs) represented by GPT family have achieved remarkable success. The characteristics of LLMs lie in their ability to accommodate a wide range of tasks through a generative approach. However, the flexibility of their output format poses challenges in controlling and harnessing the model's outputs, thereby constraining the application of LLMs in various domains. In this work, we present , an innovative toolkit designed to streamline LLM operations across diverse fields. comprises the following components: (1) a suite of task description schemas and prompt templates encompassing various NLP tasks; (2) a user-friendly, interactive process for building structured output LLM services tailored to various NLP tasks; (3) an open-source dataset for output format control, along with tools for dataset construction; and (4) an open-source model based on LLaMA3-8B-Instruct that adeptly comprehends and adheres to output formatting instructions. We anticipate this initiative to bring considerable convenience to LLM users, achieving the goal of “plug-and-play” for various applications. The components of will be progressively open-sourced at <https://github.com/cofe-ai/Sketch>.






§ INTRODUCTION



Generative pre-trained large language models (LLMs) have achieved remarkable success, with notable examples including GPT <cit.>, LLaMA <cit.>, and FLM <cit.> series. One of the key advantages of these models lies in their powerful generalization capabilities: a single model is capable of handling a diverse range of tasks.
However, accurately generating formatted outputs, such as JSON, remains challenging for LLMs because they do not always strictly follow instructions.
On the demand side, AI-driven applications urgently require the integration of structured outputs (JSON) from LLMs into their data streams. This has heightened the urgency for LLMs to produce controlled and structured outputs as demanded.

The requirement for structured outputs from LLMs can be resolved through a multitude of approaches.
In-context learning is a typical approach. It not only enhances model performance but also offers a certain degree of format control without incurring additional computational costs for model fine-tuning.
However, this approach faces challenges, such as an inability to determine when to end the generation. 
Besides, it needs long-text ability when meeting complex questions, as it relies on extensive input examples to ensure accurate decision-making. 
Moreover, tasks that require complex constraints on format and content, such as relation extraction and event extraction, pose significant difficulties for in-context learning.

Supervised fine-tuning (SFT) refers to the process of training a pre-trained model on a labelled dataset specifically tailored for a particular task.
Although SFT can enhance performance on specific tasks and has generalization capabilities, its ability to control the format of the output remains unsatisfactory.
After all, the integration of LLM outputs into applications typically demands the output format that is entirely compliant with specified requirements, a feat that LLMs, proficient in “next token prediction”, are unable to ensure.
Another issue is that, to the best of our knowledge, there is a lack of open-source models and datasets specifically addressing the problem of formatted output control.
This somewhat limits the application of LLMs across various fields.

To ensure that the outputs of LLMs conform to formatting requirements, numerous decoding control tools (guidance[<https://github.com/guidance-ai/guidance>], outlines<cit.>, llama.cpp[<https://github.com/ggerganov/llama.cpp>], lm-format-enforcer[<https://github.com/noamgat/lm-format-enforcer>] ) based on regular expressions or context-free grammars (CFGs) have been developed. 
These tools first convert the user's requirements for output format into formal languages. Under the constraints of these formal languages, these models could decode responses that meet the formatting requirements. More importantly, as these tools are involved in the decoding process of the model, they could potentially impair the model's performance<cit.>, especially if the model itself is not adept at generating structured outputs.
To address those issues, an open-source model that excels in generating structured responses according to requirements, along with a framework for streamlining various LLM-based operations, holds significant value.

In this work, we introduce , a toolkit designed to assist users in effectively operating LLMs and generating results in their expected format.
The core idea of is as follows: targeting on various NLP tasks, we establish a collection of task description schema, within which users can delineate their own tasks, including task objectives, labelling systems, and most critically, the specifications for the output format. An LLM can then be deployed out of the box to handle these unfamiliar tasks, ensuring the correctness and conformity of the output format. This approach not only streamlines the process for users but also enhances the reliability and precision of the model's outputs, making it a versatile and robust solution for a wide array of NLP applications.

The main contributions are as follows:

    
  * We propose , an innovative operating framework simplifying the process for LLM users, enabling “plug-and-play” functionality for task-specific applications with predefined schemas. The proposed makes it easier to instantiate and manage NLP tasks.
    
    
  * To optimize the performance within framework, we build a dataset and conduct model fine-tuning based on LLaMA3-8B-Instruct, ensuring superior task handling and output consistency. 
    Both the dataset and fine-tuned model will soon be made available to the public.

    
  * By integrating constrained decoding frameworks, ensures precise control over the model's output format, enhancing the reliability and precision of outcomes, and facilitating direct application of large models in industry settings.





§ ARCHITECTURE





is designed to enable controlled formatting and easy interaction with LLMs. In this section, we detail the architecture of and how to use it easily. Figure <ref> illustrates the concepts and internal workflow of . The workflow consists of four steps: schema selection, task instantiation, prompt packaging, and generation. In practical applications, the complex aspects of this process are transparent to the user.

First, users are guided to choose the appropriate schema from a predefined set that aligns with the specific NLP task requirements. A schema, in essence, is a class (or a JSON Schema [<https://json-schema.org>] in practice) that standardizes the user's description of tasks.
Second, in the task instantiation phase, users populate the chosen schema with task-specific details such as description, label set, choice type, and output format, resulting in a task instance(in JSON format) that adheres to the corresponding schema.
Third, based on the task instance, the prompt packaging step involves automatically converting the task input into a structured prompt tailored for LLM interaction.
Last, during generation, can not only infer the model to get the anticipated response but also optionally integrate the lm-format-enforcer, a control architecture that constrains LLM outputs to comply with the specified output format.





 §.§ Schema Selection


Schema is the bridge between tasks and LLMs. 
It outlines a descriptive framework for each kind of task based on the task-specific characteristics.
A schema can be represented by either a Pydantic model or a JSON Schema.
When customizing a specific task, users are advised to select the most appropriate schema and instantiate the task within its constraints.
This process can be achieved through a Python API, and we also provide a more intuitive interactive method in the form of filling out a form generated by based on the schema.
To date, as the initial phase of 's development, we have experimentally built a set of schemas for tasks, including over ten subcategories under the three main categories of text classification, text generation, and information extraction, as shown in Table<ref>.
A selection of the schemas we have crafted is showcased in Appendix <ref>. 
For an extensive view of the task schemas available, please visit our project repository at <https://github.com/cofe-ai/Sketch>





 §.§ Task Instantiation

We define Task Instance as a standardized description of a particular task within the constraints of the schema it belongs to, and the process of creating it by the user is referred to as Task Instantiation. A task instance typically includes the following basic fields:



Task specification fields delineates the task, which may encompass the “taskDesc” field detailing the task's purpose, along with the “labelSet” and “choiceType” fields that respectively define the classification schema and the number of options. We establish different required fields for various tasks.

Output format field specifies and constrains the format that users expect the model to output. We choose JSON schema as the descriptive language. This field serves a dual function: it is integrated into the prompt to direct the model's output, and it is also converted into a decoding control mechanism in the form of regular expressions, typically enforced through finite state machines (FSMs). This strategy intervenes in the model's decoding process to guarantee that the output is 100% compliant in form with the users' expectations.

A comprehensive illustration of task instances, replete with intricate details and concrete examples, can be found in Appendix <ref>.



 §.§ Prompt Packaging

The process of packaging an instantiated task and input is crucial for ensuring LLMs understand the task requirements and process the input correctly. This step involves combining the structured task description with the user's input data into a format that is optimized for interaction with the LLM.

Input Integration. The user's input, whether it be a common text snippet or any other form of information relevant to the task, will be integrated into the prompt most intuitively. This integration is guided by a prompt template associated with the schema, ensuring that the input is presented in a manner that is coherent and comprehensible to LLMs.

As shown in Figure <ref>, for a NER task, the packaged prompt might include [Task Description], [Label Architecture], [Output Format (Json Schema)], and [Input Data] to be processed. This ensures that the LLM understands the task criteria and outputs the result in the desired format.




 §.§ Generation


The final step in the workflow of involves the interaction with LLMs to generate the desired output.
is able to generate the expected response directly with a good performance. Besides, there are some more precise control methods. Throughout this process, we ensure that the model's output conforms to the required format from two perspectives.

Constrained Generation. Considering that even with meticulous fine-tuning, LLMs cannot guarantee 100% accuracy in output format, we integrate a mature decoding control framework, lm-format-enforcer. It employs CFG to ensure that the model’s responses align perfectly with the predefined output format.
Simultaneously, recognizing that any constraints to the decoding process may impact the model's performance, this strict output format control is made optional in .

Output Validation. Given that not all JSON Schema properties are supported by decoding control frameworks, the output produced by the LLM cannot be assured to adhere to the constraints of the specified output format. To ensure compliance with the expected format, we employ the jsonschema tool[<https://github.com/python-jsonschema/jsonschema>] for validation. For outputs that do not meet the expected format, we take measures such as resampling or directly throwing exceptions. 

By following these detailed steps, ensures that users can effectively utilize LLMs for a variety of NLP tasks, with the assurance that the outputs will be both accurate and in the correct format. This streamlined process makes it easier for users to interact with complex models and harness their power for practical applications.




 §.§ Code Example

Listing <ref> demonstrates the basic usage of through a simple named entity recognition (NER) task. is still under development prior to its release, and the APIs may change at any time.


    frame=lines,
    framesep=2mm,
    baselinestretch=1.2,
    fontsize=


listingExample of 's Usage 
 
[autogobble,frame=none, bgcolor=bggray]python
import llm_sketch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("CofeAI/Sketch-8B", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("CofeAI/Sketch-8B")

my_ner_task = llm_sketch.schemas.NER(
    taskDesc="Extract the named entities from the given text.",
    entityTypes=[
        "name": "person", 
        "name": "organization", 
        "name": "location"
    ],
    outputFormat=
        "type": "array",
        "items": 
            "type": "object",
            "properties": 
                "name": "type": "string", "description": "the entity name",
                "entity_type": 
                    "type": "string",
                    "description": "entity type",
                    "enum": ["person", "organization", "location"],
                ,
            ,
            "required": ["name", "entity_type"],
        ,
    ,
)

inputs = [
    "Kamala Harris pledges 'new way forward' in historic convention speech"
]
for inpt in inputs:
    ner_res = llm_sketch.generate(model, tokenizer, inpt, my_ner_task, strict=True)
    print(ner_res)
# ['name': 'Kamala Harris', 'entity_type': 'person']








§ -8B FINE-TUNING APPROACH


We fine-tune LLaMA3-8B-Instruct to enhance the model's capability to generate structured data that adheres to JSON schema constraints across a variety of tasks. Our training process emphasizes two key aspects: ensuring strict adherence to the specified JSON schema constraints in the model's outputs and fostering robust generalization across various tasks. To achieve these goals, we carefully design a specialized fine-tuning dataset.



 §.§ Data Preparation

The capability of the model to adhere to formats and its ability to understand and tackle tasks are distinct attributes. To enhance these aspects, we have constructed two targeted datasets: NLP task data and schema following data. The primary objective of NLP task data is to enable models to learn how to tackle NLP tasks. However, considering the limitations in output format diversity of manually curated fine-tuning data for NLP tasks, we propose the automated construction of schema following data to enhance the model's adherence to the output format schema.

NLP Task Data.
We assemble a comprehensive collection of over 20 datasets, encompassing more than ten subcategories within three primary domains: text classification, text generation, and information extraction. Through the meticulous design of output formats for each dataset, we construct a task instance set of size 53. Among them, 37 task instances are dedicated to training, while the remainder are reserved for evaluation.


Schema Following Data.
To ensure the diversity of JSON schemas, we generated 10,000 JSON schemas with widths and depths within 5 with a random schema generation method. Then, we utilized LLaMA3-8B-Instruct, under the constraint of a decoding control tool, to generate JSON instances that conform to the schemas. 
Following the patterns of NLP task data, we designed a task that involves selecting values from a randomly generated list of given values to construct JSON objects that match specific schemas.
Finally, we constructed 20,000 pieces of fine-tuning data for this task by modifying the values in the JSON instances generated by LLaMA3-8B-Instruct.



 §.§ Fine-tuning Method


Reinforcement learning is one of the popular ways to tune the LLMs. LeCun holds the opinion, “I do favor MPC over RL”[<https://x.com/ylecun/status/1827787323108393027>]. We have a similar opinion so we use the easy fine-tuning methods under data-driven. Indeed, it doesn't mean reinforcement learning is useless, but it could be used in the following steps such as resort. Generating valid outputs that conform to the JSON Schema is not simply a matter of mimicking formats, it necessitates a thorough comprehension of the schema's descriptions. Consequently, data adhering to the schema is essential for enhancing the model's ability.


The training objective of -8B considers two aspects: enhancing the model's adherence to format and improving its NLP task performance. To this end, we use the proposed mixed dataset comprising NLP task data and schema following data for fine-tuning. 
The inclusion of NLP task data markedly boosts the model’s capabilities in handling NLP tasks while the schema following data is crucial for enhancing the model’s adherence to various output format requirements.



We use fine-tuning method to optimize the proposed model, the objective ℒ(θ) could be formatted as:

    ℒ(θ) = -∑^m_t=1 logP_θ(ŷ_̂t̂=y_t|y_1:t-1,X)

where X = {x_1, x_2, …, x_n} represents an input sequence of length n, which is the constructed prompt. Y = {y_1, y_2, …, y_m} is the label of the generated sequence of length m, and Ŷ = {ŷ_1, ŷ_2, …, ŷ_m} is the actual output of the model. Note that both Y and Ŷ exclude the prompt and consist only of the response. θ denotes the model parameters, and P_θ represents the conditional probability under the parameters θ.

Each sample consisting of X and Y is sampled from a carefully constructed mixed dataset. The optimal fine-tuning effect is achieved by appropriately balancing the ratio of NLP task data to schema-following data in the mixed dataset.




§ EXPERIMENTS







 §.§ Experiment Settings



In this section, we validate the model's generalization capabilities through experiments and discuss the effectiveness and optimal configuration of our fine-tuning data. 



  
Experiment Data Settings.
We use publicly available NLP task datasets (See Appendix <ref> for details) for experiments. For each dataset, we carefully construct different task instances, expanding a single dataset into multiple experimental datasets with varying outputFormat and other task-related parameter configurations. To validate different hypotheses, we selectively exclude some data from the training set to create test datasets. These test datasets include three types: (1) the output format not seen in the training set while other output formats from the same dataset are included, (2) the entire dataset is not present in the training set, and (3) the entire tasks are not included in the training set. 




  
Fine-tuning Settings.
We experiment on LLaMA3-8B-Instruct since it has strong foundational capabilities. We fine-tune the model for 8 epochs with a global batch size of 128, setting the learning rate to 1e-6 and weight decay to 0.1. The learning rate is decayed to 0 using a linear schedule. We select the best checkpoint from the model at the end of every epoch.




  
Evaluation Methods.
To comprehensively evaluate the model's schema adherence and NLP task performance, we assess from two perspectives:

    
  * We define a metric to assess the model's ability to produce outputs that conform to the outputFormat: Legal Output Ratio. First, we determine whether the model's output can be converted into a JSON object; if not, the output is considered invalid. Next, we check if the JSON object meets the outputFormat requirements; otherwise, it is considered invalid. The legal output ratio is calculated by dividing the number of valid outputs by the total number of test samples.
    
  * To evaluate NLP task performance, we employ traditional metrics like F1-score or accuracy, tailored to the specific requirements of each task.







 §.§ Comparison with Baselines

To evaluate generalization, we fine-tune -8B-w.o.-ner with a partially removed dataset and benchmark it against mainstream models, including GPT-4o, DeepSeek, and ChatGLM. Using identical prompts across models, we gather results via API and assess performance. We also compare -8B-w.o.-ner with the original LLaMA3-8B-Instruct (local inference). Additionally, we evaluate DeepSeek's one-shot results and GPT-4o's constrained decoding. -8B-w.o.-ner and LLaMA-8B-Instruct use FSM and CFG constraints for decoding. The comparison covers three dataset types: (1) unknown format, with output formats absent in training data, (2) unknown domain, with datasets from untrained domains, and (3) unknown task, focusing on task types not covered during training. NER is the test task for the Unknown Task category.





Schema Adherence Comparison.
Table <ref> illustrates notable differences in schema adherence among baseline models under unconstrained output conditions. For simpler formats like S10T8 and HOTEL, LLaMA3-8B-Instruct achieves nearly 100% on legal output ratio but fails completely on 20NEWS. Across most datasets, its legal output ratio ranges from 50% to 75%, averaging 64.9%. In contrast, -8B-w.o.-ner achieves an average legal output ratio of 96.2% under unconstrained conditions, with its lowest performance on CNL03 still at 83.8%. This demonstrates -8B-w.o.-ner's strong generalization in format adherence.

Performance Comparison.
We compare with LLaMA3-8B-Instruct to assess training effectiveness and with mainstream models to evaluate performance level:

1. vs LLaMA3-8B-Instruct. Table <ref> shows that -8B-w.o.-ner consistently outperforms LLaMA3-8B-Instruct under the same decoding strategy, both on individual subsets and in average scores. Furthermore, the unconstrained -8B-w.o.-ner surpasses LLaMA3-8B-Instruct across all decoding strategies. The results indicate that the fine-tuning method enhances NLP task performance and demonstrates strong generalization to unknown output formats and tasks.


2. vs Mainstream Models. Comparing -8B-w.o.-ner with mainstream models like DeepSeek, ChatGLM, and GPT-4o on unknown format datasets, -8B-w.o.-ner significantly outperforms all, achieving nearly 100% legal output ratio on 20NEWS where others struggle (GPT-4o below 50%). On unknown domain datasets, it performs similarly to DeepSeek and GPT-4o but surpasses ChatGLM. However, its smaller model size leads to some limitations on unknown task datasets compared to larger models.


Constrained Decoding Evaluation.
The analysis also reveals that FSM and CFG constraints do not consistently produce the expected outcomes. FSM constraints result in lower task evaluation scores for both -8B and LLaMA3-8B-Instruct. While CFG constraints improve overall average scores, they fail to enhance task evaluation scores on datasets with hard output formats (20NEWS), despite increasing the legal output ratio. This suggests that current constrained decoding methods are not yet consistently reliable for real-world NLP tasks.



 §.§ Generalization Capability Analysis





  
Output Format Generalization Capability.

We first evaluate -8B's generalization capability across different output formats within the same dataset.
As shown in the “Unknown Format” column of Table <ref>, the output formats of the two datasets (S10T8 and 20NEWS) used for evaluation are not in -8B's training set.
We can observe that in S10T8, both LLaMA3-8B-Instruct and -8B achieve high precision (0.997 and 0.982) in adhering to the required output format, which is likely due to the format simplicity. 
For 20NEWS, due to the complex format, LLaMA3-8B-Instruct is completely unable to follow the required output format. Surprisingly, despite not being trained in this specific format, -8B shows an impressive ability to follow output format. This demonstrates -8B's generalization ability on unseen formats within the trained dataset.





  
Domain Generalization Capability.

Further, we evaluate -8B's cross-domain generalization capability within the same task (NER). This is crucial for models' application in various scenarios from various users. We continue to evaluate -8B on two tasks: aspect-level sentiment analysis, and text topic classification. We construct two datasets that are untrained and completely different from the domains of -8B's training datasets. The results in Table <ref> column “Unknown Domain” show that -8B significantly outperforms LLaMA3-8B-Instruct on these two datasets (domains), both in terms of adherence to formatting requirements and NLP task F1/Accuracy. It is important to note that -8B has never encountered data from these three domains during training. 
This illustrates a fact: -8B is capable of enhancing its performance across different domains within a task by training on data associated with specific domains (or taxonomies). 



  
Task Generalization Capability.
Ultimately, we evaluate the ability to generalize across tasks. This ability is known as the most formidable aspect of generalization. While we can endeavour to build an extensive array of NLP task categories, the spectrum of potential tasks is infinite. As such, LLM users across a myriad of sectors undoubtedly desire a model that can extend its reach to cover their unique and unconventional task needs. This is why we present the evaluation results of -8B-w.o.-ner in Table <ref>. We completely exclude the NER datasets from the training set and evaluate how the output format following capabilities of -8B-w.o.-ner improve on NER tasks.
Remarkably, -8B-w.o.-ner demonstrates significant improvement in the two NER datasets, with L.O.R. increasing from 0.520 to 0.939 and from 0.645 to 0.968, respectively.
Consequently, it can be concluded that for an unfamiliar NLP task, -8B is likely a superior choice compared to LLaMA3-8B-Instruct, even though it has not been trained on such tasks.




 §.§ Data Configuration Experiment


Fine-tuning data is central to this work. We analyze how data proportion and scale affect model performance. The evaluation focuses on the model's results on a test set with seven tasks: three with unseen output formats, three from unseen domains, and one entirely new task.




Data Proportion.
Different sampling proportion affects the performance of pretraining foundation models. Similar to this phenomenon, the sampling proportion of schema following data leads to a decline in task performance.

To assess the effectiveness and configuration of NLP task data and schema following data, we conduct experiments using a fixed 20k dataset with various proportions, including a setup without schema following data. Performance is evaluated on the test set, with results shown in Table <ref>.

From the table, we observe that the schema following data proportion is positively correlated with the legal output ratio. Schema following data significantly enhances the model's ability to follow output formats. However, when the schema following data proportion exceeds 25%, performance on the test set declines from 0.688 to 0.655, indicating that excessive schema following data negatively impacts task performance. Therefore, we determine that a 7:1 ratio of Task Data to schema following data is optimal.




Data Volume.
We conduct experiments to analyze the impact of data size on results. Four fine-tuning datasets with 10k, 20k, 30k, and 40k samples (with a 7:1 ratio of Task Data to schema following data) are used to train the model, which are then evaluated on the same test set.

As shown in Table <ref>, the best legal output ratio 0.697 is achieved with the 30k dataset. Increasing the data size to 40k leads to a noticeable decline in both performance and legal output ratio. This suggests that more fine-tuning data does not always yield better results. We ultimately select the 30k dataset for training the -8B model.




§ RELATED WORK



Significant advancements have been made in the realm of format-constrained generation for LLMs. We roughly divide these methods into three categories: pre-generation tuning, in-generation control, and post-generation parsing.



  
Pre-generation Tuning.
Pre-generation tuning encompasses a suite of techniques designed to fine-tune the behaviour of LLMs before the actual text-generation process begins. This approach involves modifying the model's training data<cit.> or prompts<cit.> to align more closely with the specific format constraints required by the task at hand.



  
In-generation Control.

There are numerous frameworks dedicated to intervening in the decoding process of LLMs to control the permissible range of the next token, ensuring that the output of the LLM meets the format requirements. The predominant control strategies employed include JSON Schema (Jsonformer[<https://github.com/1rgs/jsonformer>], lm-format-enforcer and outlines), regular expression (guidance, lm-format-enforcer
 and outlines) and context-free grammar
 (llama.cpp). Although these methods typically ensure high accuracy in response format, they often lead to a decrease in the usefulness of the responses<cit.>, which is one of the starting points for the work presented in this paper.






  
Post-generation Parsing.
This category involves techniques that parse the output of LLMs after generation to ensure it conforms to specific formats. These methods often rely on post-processing algorithms to refine the raw output into a structured format. Guardrails[<https://github.com/guardrails-ai/guardrails>] is a framework of this kind, designed to enforce constraints on the output of LLMs by filtering or modifying the generated text to ensure it adheres to predefined guidelines or specifications. 




§ CONCLUSIONS AND FUTURE WORK




In this work, we propose to simplify and optimize the applications of LLMs. Using a schema-based approach, can tackle the challenges in structured output generation and model generalization. Key contributions include the schema architecture for task description, data and model fine-tuning for improved performance, and the integration of a constrained decoding framework for precise output management. 
Experimental results not only demonstrate the enhanced capability of the fine-tuned -8B in adhering to output formats but also validate the effectiveness of the fine-tuning data we build, particularly the schema following data.

Future work involves expanding task categories, optimizing model performance, lowering entry barriers, and exploring new applications in diverse domains. 's innovative approach and ongoing development promise to drive advancements in LLM applications and unlock new possibilities for harnessing the power of LLMs.



§ ACKNOWLEDGMENTS

This work is supported by the National Science and Technology Major Project (No. 2022ZD0116300) and the National Science Foundation of China (No. 62106249).


10

    gpt3
    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
    Language models are few-shot learners.
    Advances in neural information processing systems, 33:1877–1901, 2020.
    
    DBLP:conf/nips/BrownMRSKDNSSAA20
    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
    Language models are few-shot learners.
    In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
    
    DBLP:journals/corr/abs-2003-04807
    Iñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic.
    Efficient intent detection with dual sentence encoders.
    CoRR, abs/2003.04807, 2020.
    
    rte1
    Ido Dagan, Oren Glickman, and Bernardo Magnini.
    The PASCAL recognising textual entailment challenge.
    In Joaquin Quiñonero Candela, Ido Dagan, Bernardo Magnini, and Florence d'Alché-Buc, editors, Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers, volume 3944 of Lecture Notes in Computer Science, pages 177–190. Springer, 2005.
    
    Llama3
    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo
      Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al.
    The llama 3 herd of models.
    CoRR, abs/2407.21783, 2024.
    
    DBLP:conf/acl/FitzGeraldHPMRS23
    Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gökhan Tür, and Prem Natarajan.
    MASSIVE: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages.
    In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 4277–4302. Association for Computational Linguistics, 2023.
    
    DBLP:conf/sigsoft/GranoSMVCP17
    Giovanni Grano, Andrea Di Sorbo, Francesco Mercaldo, Corrado Aaron Visaggio, Gerardo Canfora, and Sebastiano Panichella.
    Android apps and user feedback: a dataset for software evolution and quality improvement.
    In Federica Sarro, Emad Shihab, Meiyappan Nagappan, Marie Christin Platenius, and Daniel Kaimann, editors, Proceedings of the 2nd ACM SIGSOFT International Workshop on App Market Analytics, WAMA@ESEC/SIGSOFT FSE 2017, Paderborn, Germany, September 5, 2017, pages 8–11. ACM, 2017.
    
    DBLP:conf/icml/GreeneC06
    Derek Greene and Padraig Cunningham.
    Practical solutions to the problem of diagonal dominance in kernel document clustering.
    In William W. Cohen and Andrew W. Moore, editors, Machine Learning, Proceedings of the Twenty-Third International Conference (ICML 2006), Pittsburgh, Pennsylvania, USA, June 25-29, 2006, volume 148 of ACM International Conference Proceeding Series, pages 377–384. ACM, 2006.
    
    DBLP:journals/corr/abs-1911-10422
    Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz.
    Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals.
    CoRR, abs/1911.10422, 2019.
    
    LANG1995331
    Ken Lang.
    Newsweeder: Learning to filter netnews.
    In Armand Prieditis and Stuart Russell, editors, Machine Learning Proceedings 1995, pages 331–339. Morgan Kaufmann, San Francisco (CA), 1995.
    
    freelm
    Xiang Li, Xin Jiang, Xuying Meng, Aixin Sun, and Yequan Wang.
    Freelm: Fine-tuning-free language model.
    CoRR, abs/2305.01616, 2023.
    
    flm101b
    Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Xuying Meng, Siqi Fan, Peng Han, Jing Li, Li Du, Bowen Qin, et al.
    Flm-101b: An open llm and how to train it with $100 k budget.
    arXiv preprint arXiv:2309.03852, 2023.
    
    teleflm
    Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Chao Wang, Xinzhang Liu, Zihan Wang, Yu Zhao, Xin Wang, Yuyao Huang, Shuangyong Song, Yongxiang Li, Zheng Zhang, Bo Zhao, Aixin Sun, Yequan Wang, Zhongjiang He, Zhongyuan Wang, Xuelong Li, and Tiejun Huang.
    Tele-flm technical report.
    CoRR, abs/2404.16645, 2024.
    
    DBLP:conf/nlpcc/LiLPCPWLZ20
    Xinyu Li, Fayuan Li, Lu Pan, Yuguang Chen, Weihua Peng, Quan Wang, Yajuan Lyu, and Yong Zhu.
    Duee: A large-scale dataset for chinese event extraction in real-world scenarios.
    In Xiaodan Zhu, Min Zhang, Yu Hong, and Ruifang He, editors, Natural Language Processing and Chinese Computing - 9th CCF International Conference, NLPCC 2020, Zhengzhou, China, October 14-18, 2020, Proceedings, Part II, volume 12431 of Lecture Notes in Computer Science, pages 534–545. Springer, 2020.
    
    maas-EtAl:2011:ACL-HLT2011
    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.
    Learning word vectors for sentiment analysis.
    In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
    
    DBLP:conf/emnlp/NarayanCL18
    Shashi Narayan, Shay B. Cohen, and Mirella Lapata.
    Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.
    In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 1797–1807. Association for Computational Linguistics, 2018.
    
    GPT-4
    OpenAI.
    GPT-4 technical report.
    CoRR, abs/2303.08774, 2023.
    
    corona
    Ashish Patil.
    Medical ner.
    Kaggle, 2020.
    
    DBLP:conf/semeval/PontikiGPMA15
    Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos.
    Semeval-2015 task 12: Aspect based sentiment analysis.
    In Daniel M. Cer, David Jurgens, Preslav Nakov, and Torsten Zesch, editors, Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2015, Denver, Colorado, USA, June 4-5, 2015, pages 486–495. The Association for Computer Linguistics, 2015.
    
    DBLP:conf/semeval/PontikiGPPAM14
    Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar.
    Semeval-2014 task 4: Aspect based sentiment analysis.
    In Preslav Nakov and Torsten Zesch, editors, Proceedings of the 8th International Workshop on Semantic Evaluation, SemEval@COLING 2014, Dublin, Ireland, August 23-24, 2014, pages 27–35. The Association for Computer Linguistics, 2014.
    
    DBLP:conf/conll/SangM03
    Erik F. Tjong Kim Sang and Fien De Meulder.
    Introduction to the conll-2003 shared task: Language-independent named entity recognition.
    In Walter Daelemans and Miles Osborne, editors, Proceedings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with HLT-NAACL 2003, Edmonton, Canada, May 31 - June 1, 2003, pages 142–147. ACL, 2003.
    
    tam2024let
    Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen.
    Let me speak freely? a study on the impact of format restrictions on performance of large language models.
    arXiv preprint arXiv:2408.02442, 2024.
    
    DBLP:journals/eswa/TanZ08
    Songbo Tan and Jin Zhang.
    An empirical study of sentiment analysis for chinese documents.
    Expert Syst. Appl., 34(4):2622–2629, 2008.
    
    llama1
    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.
    Llama: Open and efficient foundation language models.
    CoRR, abs/2302.13971, 2023.
    
    llama-2
    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov,
      and Thomas Scialom.
    Llama 2: Open foundation and fine-tuned chat models.
    CoRR, abs/2307.09288, 2023.
    
    DBLP:conf/iclr/WangSMHLB19
    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.
    GLUE: A multi-task benchmark and analysis platform for natural language understanding.
    In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
    
    DBLP:conf/nips/Wei0SBIXCLZ22
    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou.
    Chain-of-thought prompting elicits reasoning in large language models.
    In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022.
    
    willard2023efficient
    Brandon T Willard and Rémi Louf.
    Efficient guided generation for llms.
    arXiv preprint arXiv:2307.09702, 2023.
    
    yao2024opendomainimplicitformatcontrol
    Yiqun Yao, Wenjia Ma, Xuezhi Fang, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Jing Li, Aixin Sun, and Yequan Wang.
    Open-domain implicit format control for large language model generation, 2024.
    
    amazon_polarity
    Xiang Zhang, Junbo Jake Zhao, and Yann LeCun.
    Character-level convolutional networks for text classification.
    In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649–657, 2015.
    
    DBLP:conf/emnlp/ZhangZCAM17
    Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning.
    Position-aware attention and supervised data improve slot filling.
    In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 35–45. Association for Computational Linguistics, 2017.
    
    zhou2023controlledtextgenerationnatural
    Wangchunshu Zhou, Yuchen Eleanor Jiang, Ethan Wilcox, Ryan Cotterell, and Mrinmaya Sachan.
    Controlled text generation with natural language instructions.
    In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 42602–42613. PMLR, 2023.
    
    DBLP:conf/iclr/Zhou00CP24
    Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon.
    Universalner: Targeted distillation from large language models for open named entity recognition.
    In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
    
    
    









§ SCHEMA EXAMPLES











§ TASK INSTANCE EXAMPLES












§ NLP TASK DATASETS


In this appendix, we provide a detailed description of the datasets utilized in this paper. These datasets are categorized into one of the following task categories:


  * Information extraction Information extraction (IE) encompasses the task of discerning and extracting structured information from unstructured and/or semi-structured machine-readable documents. This category includes various sub-tasks such as relation extraction, named entity recognition, event extraction, and aspect-level sentiment analysis. The following datasets are utilized to facilitate these tasks: 

    
  * Relation extraction: SemEval-2010 Task 8<cit.>, TACRED<cit.>;
    
  * Named entity recognition: CoNLL-2003<cit.>, UniversalNER<cit.>, Medical NER<cit.>;
    
  * Aspect-level sentiment analysis: SemEval-2014 Task 4<cit.>; SemEval-2015 Task 12<cit.>; 
    
  * Event extraction: DuEE<cit.>;



  * Text classification Text classification is the task of assigning predefined categories to text documents. It encompasses a wide range of tasks, such as sentiment analysis, topic classification, intent recognition, sentence similarity, We use the following datasets:

    
  * Sentiment analysis: APP_REVIEWS<cit.>, ChnSentiCorp<cit.>, IMDB<cit.>;
    
  * Topic classification: 20 Newsgroups<cit.>, AG News<cit.>, BBC News<cit.>, DBPedia<cit.>;
    
  * Intent recognition: MASSIVE<cit.>, BANKING77<cit.>;
    
  * Sentence similarity(also known as paraphrase detection): QQP<cit.>;
    
  * Natural language inference: RTE<cit.>;



  * Text generation Text generation involves creating text from scratch or completing partial texts based on given prompts. This task is essential for applications such as chatbots, translation, and summarization. We use the following datasets:

    
  * Summarization: xsum<cit.>;
    
  * Translation: Replete-AI/Multi-lingual_Translation_Instruct[<https://huggingface.co/datasets/Replete-AI/Multi-lingual_Translation_Instruct>];
    
  * Dialog: shibing624/sharegpt_gpt4[<https://huggingface.co/datasets/shibing624/sharegpt_gpt4>];





